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abstract 


The  purpose  of  this  study  was  to  investigate  the  effect  of  instructional  method 
(case-based  approach  vs.  systematic  approach)  and  collaborative  group  size  on  learners' 
ability  to  solve  ill-defined  problems  in  a  web-based  environment.  Working  in  teams  of 
two  (dyads)  or  teams  of  four  (quads),  the  participants  learned  how  to  solve  problems 
using  case-studies  (case-based  approach)  or  they  learned  a  four  step  problem-solving 
process  (systematic  approach).  The  participants  first  worked  through  a  web-based 
instructional  program  that  taught  them  the  problem-solving  approach.  Then,  they  applied 
the  problem-solving  approach  to  solve  realistic  problem  scenarios.  Results  indicated  that 
participants  who  worked  in  dyads  performed  significantly  better  than  participants  who 
worked  in  quads  on  one  of  two  problem  scenarios.  The  data  did  not  indicate  a  significant 
difference  by  instructional  method.  Participants  who  worked  in  quads  spent  significantly 
more  time  on  the  learning  program  than  participants  working  in  dyads.  Analysis  of  time 
spent  solving  problems  revealed  that  participants  who  used  the  systematic  approach  spent 
more  time  solving  the  problems  than  did  participants  in  the  case-based  approach.  All 
treatment  groups  had  positive  attitudes  toward  working  with  others,  the  instructional 
materials,  and  applicability  of  problem-solving  skills  to  other  settings.  However, 
working  collaboratively  using  a  computer-mediated  medium  was  the  lowest  rated  item  in 
the  attitude  survey,  indicating  participants  did  not  enjoy  collaborating  using  a  computer. 
Analysis  of  the  communications  between  team  members  indicated  that  95%  of  the 
communications  that  took  place  were  related  to  the  problem-solving  task.  Implications 
for  the  implementation  of  computer-mediated  collaboration  in  distance  learning  are 

discussed. 
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Introduction 


Problem  solving  is  regarded  as  one  of  the  most  important  cognitive  activities  in 
everyday  life  and  a  primary  goal  of  the  education  process  (Jonassen,  2000;  Phye,  2001). 
How  to  design  effective  instructional  programs  that  help  learners  acquire  the  skills  to 
solve  ill-defined  problems  has  become  a  focus  of  the  instructional  technology  field  in 
recent  years.  According  to  Jonassen  (2000),  ill-defined  problems  are  characterized  by  a 
lack  of  knowledge  of  all  elements  of  the  problem,  multiple  solutions  or  solution  paths, 
multiple  criteria  for  evaluating  solutions  and  a  need  for  the  learner  to  make  judgments 
and  express  personal  opinions  or  beliefs. 

Some  critics  argue  that  current  instructional  design  models  are  not  well  suited  to 
teach  the  skills  necessary  to  solve  ill-defined  problems  (Van  Merrienboer,  Clark,  Moore 
&  de  Crook,  2002).  To  fill  this  apparent  gap,  several  problem-solving  instructional 
strategies  based  on  current  cognitive  and  learning  theories  have  been  advanced.  These 
strategies  emphasize  solving  realistic  problems  in  authentic  contexts  (Oliver  & 
Harrington,  2000;  Bastiaens  &  Martens,  2000). 

The  following  paragraphs  present  a  review  of  the  literature  on  case-based  learning 
and  the  systematic  approach  to  problem  solving.  Also,  the  relationship  between 
collaborative  learning  and  problem  solving  is  explored  and  a  review  of  the  research  on 
the  effect  of  group  size  on  performance  is  provided. 
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Case-Based  Approach  to  Problem  Solving 

Proponents  of  case-based  learning  describe  it  as  a  powerful  instructional  approach 
that  is  engaging  and  that  leads  to  sustained  and  transferable  learning  of  problem-solving 
skills  (Mergendoller,  Bellisimo,  &  Maxwell,  2000;  Stepien  &  Gallagher,  1993).  Case- 
based  learning  uses  authentic,  complex  problems  as  the  impetus  for  learning  and  fosters 
the  acquisition  of  both  disciplinary  knowledge  and  problem-solving  skills  (Bligh,  1995; 
Edens,  2000;  Flynn  &  Klein,  2001;  Levin,  1995).  Because  of  its  potential  to  enhance 
knowledge  acquisition,  case-based  learning  has  become  a  popular  method  to  deliver 
classroom  instruction  in  education,  and  it  has  been  used  in  a  variety  of  academic 
environments  (Edens,  2000;  Flynn  &  Klein  2001,  Kinzie,  Hrabe  &  Larsen,  1998, 
Shulman,  1 992).  Julian,  Kinzie  and  Larsen  (2000)  propose  that  case-based  learning  is 
more  effective  than  didactic  teaching  methods  because  it  more  accurately  represents  the 
complexity  and  ambiguity  of  real-life  problems  and  it  provides  a  means  for  allowing 
learners  to  develop  the  kind  of  problem-solving  strategies  that  practicing  professionals 
use. 

Van  Merrienboer  et  al.  (2002)  identify  case-based  learning  in  the  4 
Components/Instructional  Design  (4C/ID)  model  as  a  scaffolding  technique  that  supports 
the  learner  at  the  beginning  of  complex  tasks.  According  to  Van  Merrienboer  et  al.,  case 
studies  are  product-oriented  support  for  complex  learning  that  provide  the  learner  with  a 
given  state,  a  desired  goal  state,  and  the  solution  to  the  problem.  They  theorize  that  this 
scaffolding  will  help  the  learner  develop  mental  models  from  the  given  examples  that 
will  aid  in  the  solution  of  the  same  type  of  complex  problems. 


Systematic  Approach  to  Problem  Solving 

Another  technique  that  can  be  used  to  solve  complex  tasks  is  that  of  a  systematic 
approach  to  problem  solving.  Van  Merrienboer  et  al.  (2002)  describes  the  systematic 
approach  as  a  strategy  that  uses  heuristics  and/or  rules-of-thumb,  which  can  be  applied  to 
solve  a  complex  task.  Gagne  (1985)  describes  problem  solving  as  an  extension  of  both 
rule  learning  and  schema  learning.  He  argues  that  problem  solving  is  not  just  the 
application  of  rules  to  achieve  a  goal  or  solve  a  problem,  but  it  is  also  a  process  that 
yields  new  learning.  When  a  learner  is  faced  with  a  problem  situation,  Gagne  (1985) 
argues,  the  learner  will  recall  previously  learned  rules  in  an  attempt  to  find  a  solution.  In 
applying  the  rules  and  finding  a  solution,  the  learner  will  also  develop  a  “higher-order 
rule”  that  can  be  applied  to  problems  in  the  same  domain  or  of  similar  characteristics. 
Along  the  same  lines,  Van  Merrienboer  (in  press)  describes  the  systematic  approach  as  a 
process-oriented  learner  support  for  complex  learning  that  provides  the  learner  with  a 
cognitive  strategy  that  can  be  applied  when  the  learner  is  confronted  with  problem¬ 
solving  tasks  in  the  same  domain. 

A  systematic  approach  to  solving  problems  has  been  found  to  be  effective  in  a 
variety  of  academic  disciplines.  For  example,  a  systematic  approach  that  combines 
elements  of  clinical  reasoning  and  the  scientific  approach  is  taught  early  to  medical 
students  and  is  refined  as  the  student  gains  experience  (Benjamin  &  Hamdy,  1993).  In  a 
study  of  high-school  mathematics  students  in  the  Netherlands  that  compared  different 
approaches  to  problem  solving,  the  results  revealed  that  students  that  learned 
mathematics  through  a  heuristic  or  systematic  approach  were  able  to  use  their 
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mathematical  knowledge  to  solve  problems  better  than  students  who  used  a  discovery 
approach.  (Van  Streun,  2000). 

Collaborative  Learning  and  Problem  Solving 

The  mental  models  and  cognitive  strategies  developed  by  the  learner  to  solve 
complex  tasks  can  be  enhanced  through  the  use  of  a  collaborative  environment.  Some 
current  research  suggests  that  a  collaborative  learning  environment  can  positively  affect 
performance  on  problem  solving  tasks  (Flynn  &  Klein,  2001 ;  Johnson,  Johnson,  &  Smith, 
1991;  Mergendoller  et  al.,  1999).  Collaborative  learning  is  defined  as  "an  activity  that  is 
undertaken  by  equal  partners  who  work  jointly  on  the  same  problem  rather  than  on 
different  components  of  the  problem"  (Brandon  &  Hollingshead,  1999).  Smith  and 
McGregor  (1992)  also  defined  collaborative  learning  as  a  “joint  intellectual  effort.” 

Although  collaborative  learning  is  often  used  synonymously  with  cooperative 
learning,  there  are  some  important  differences  between  the  two  learning  strategies. 
According  to  Johnson  and  Johnson  (1988)  the  five  elements  necessary  for  a  successful 
cooperative  environment  are  positive  interdependence,  face-to-face  promotive 
interaction,  interpersonal  and  group  skills,  group  processing,  and  individual 
accountability.  In  contrast  to  cooperative  learning,  collaborative  learning  is  less 
structured  and  requires  students  to  work  more  closely  together  on  a  task.  Damon  and 
Phelps  (1989)  state  that  the  two  elements  essential  for  collaboration  and  which 
distinguish  it  from  cooperation  are  equality,  or  students  of  similar  abilities  working 
together,  and  mutuality  or  learners  working  on  the  same  problem  rather  than  different 
components  of  the  problem.  A  meta-analysis  of  the  use  of  collaborative  learning  in 
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higher  education  courses  indicated  that  collaborative  learning  promotes  higher 
achievement,  higher-level  reasoning,  more  frequent  generation  of  ideas  and  solutions, 
and  greater  transfer  of  learning  than  individual  or  competitive  learning  strategies 
(Johnson  et  al.,  1991).  Other  researchers  suggest  that  collaborative  learning  is  more 
effective  than  cooperative  learning  in  areas  requiring  higher  order  thinking  skills  such  as 
problem  solving  (Damon  &  Phelps,  1989;  Doran,  1994). 

The  research  conducted  to  examine  the  effect  of  collaboration  on  problem  solving 
supports  the  hypothesis  that  a  collaborative  learning  environment  is  well  suited  for 
solving  problems  (Spector  &  Davidsen,  2000).  In  several  studies  conducted  to  analyze 
the  impact  of  a  collaborative  environment  on  problem  solving,  collaboration  was  found  to 
improve  performance  on  complex  or  higher-order  thinking  activities  (Chang  &  Smith, 
1991;  Johnson  &  Chungh,  1999;  Mergendoller  et  al.,  1999).  In  these  studies,  learners 
appeared  to  benefit  from  the  opportunity  to  discuss  the  problem,  brainstorm  potential 
solutions  and  arrive  at  a  final  solution.  However,  these  studies  have  been  conducted  in 
face-to-face  environments.  Additional  empirical  research  is  necessary  to  indicate 
whether  the  positive  effects  of  collaborative  learning  during  problem  solving  tasks  will 
also  be  obtained  in  a  computer-mediated  collaborative  environment.  With  enrollments  in 
courses  delivered  over  the  Internet  in  the  United  States  already  at  well  over  100,000 
students  (Simonson,  Smaldino,  Albright  &  Zvacek,  2000),  it  is  important  to  confirm  that 
the  positive  effects  of  collaborative  learning  in  a  face-to-face  environment  are  also 
evident  in  a  computer-mediated  collaborative  learning  structure. 
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Computer-Mediated  Collaborative  Learning 

A  web-based  distance  education  environment  may  be  well  suited  for  the  teaching 
and  learning  of  complex  skills  such  as  problem  solving.  In  the  web-based  environment,  a 
realistic  setting  can  be  created  where  the  learner  has  access  to  a  vast  amount  of  authentic 
information  and  can  work  collaboratively  through  real-time  computer-mediated 
communications  to  solve  ill-defined  problems.  These  characteristics  of  the  web  and  a 
computer-mediated  environment  may  tend  to  facilitate  problem-based  learning. 

According  to  Laffey,  Tuper  and  Wedman  (1998),  computer-mediated  learning  on  the 
World  Wide  Web  is  suitable  for  problem-based  learning  because  it  provides  ample 
resources,  allowing  students  to  do  their  own  planning  and  present  new  forms  of 
knowledge,  which  expand  the  mechanisms  for  collaboration  and  communication.  Others 
also  argue  that  computer-mediated  collaboration  and  the  web  are  excellent  technologies 
for  case  studies  and  integrating  higher-order  learning  (Jonassen,  Prevish,  Christy  & 
Stavrulaki,  1999).  Miller  and  Miller  (2000)  also  suggest  that  the  characteristics  of  the 
Internet  environment,  which  provides  a  hyperlink  structure  with  easy  access  to  relevant 
information,  realistic  and  enhanced  media  and  synchronous  communication  capabilities, 
make  it  an  effective  learning  environment  for  complex  skills. 

However,  additional  research  into  the  effect  of  synchronous  computer-mediated 
communication  on  solving  complex  tasks  is  necessary.  According  to  Murphy  and  Collins 
(1997),  research  on  synchronous  computer-mediated  communication  has  been  limited  to 
investigations  of  the  recreational  use  of  online  chat  systems.  But  the  use  of  these 
systems  for  instructional  purposes,  and  specifically  for  problem-solving  tasks,  has  been 


explored  only  through  case  studies.  These  case  studies  support  the  hypothesis  that  the 
benefits  of  collaborative  environment  in  a  face-to-face  environment  are  also  found  in  a 
computer-mediated  environment  (Hall,  1997;  Johnston,  1996;  Naidu  &  Oliver,  1999). 

Current  research  on  computer-mediated  collaborative  learning  indicates  that  it  is 
effective  when  students  are  faced  with  higher-order  cognitive  tasks  such  as  problem 
solving  (Johnston,  1996).  In  a  short  pilot  study  on  the  use  of  computer-mediated 
collaborative  groups  in  post-compulsory  teacher  education  in  the  United  Kingdom, 
results  indicated  that  students  using  computer-mediated  communications  worked  better 
with  higher-order  cognitive  tasks  than  students  in  the  control  group  who  collaborated 
face-to-face  (Hall,  1997).  In  another  case  study  where  collaborative  learning  facilitated 
through  computer-communication  was  used,  nurse  practitioners  appeared  to  derive  more 
benefit  from  the  experience  of  their  peers  by  working  and  sharing  information  via 
computer-mediated  communications  than  a  group  of  students  who  did  not  have  access  to 
computer-mediated  communication  (Naidu,  et.al.,  1999). 

Other  research  also  indicates  that  the  quality  of  interaction  between  learners  in  a 
computer-mediated  environment  may  actually  be  better  than  interaction  in  a  face-to-face 
environment.  Findings  in  a  case  study  suggested  that  computer-mediated  groups  seemed 
to  put  more  thought  into  the  comments  they  made,  thus  providing  higher  quality 
responses  than  students  who  worked  face-to-face  (Camin,  Hall,  Quarantillo  & 
Merenstein,  2001).  Hillman  (1999)  also  found  that  the  interaction  patterns  of  computer- 
mediated  groups  resembled  thoughtful  discussions  whereas  face-to-face  interactions 
resembled  recitations.  And  in  yet  another  study  where  computer-mediated 


communications  was  compared  to  face-to-face  interactions,  findings  suggest  that  in  the 
computer-mediated  environment  there  was  a  tendency  to  share  ideas  without  the 
restraints  of  typical  social  conventions,  which  resulted  in  deeper  and  more  thoughtful 
discussions  (Kruger  &  Cohen,  1996). 

In  a  recent  study  that  compared  computer-mediated  collaboration  versus 
individual  learning,  Uribe,  Klein  and  Sullivan  (in  press)  found  that  learners  who 
collaborated  through  a  synchronous  computer-mediated  environment  to  resolve  an  ill- 
defined  problem  performed  significantly  better  than  learners  who  worked  alone.  These 
results  support  the  hypothesis  that  computer-mediated  collaboration  has  a  positive  effect 
on  performance  when  resolving  ill-defined  problems.  The  results  also  showed  that, 
overall,  learners  preferred  to  work  collaboratively  versus  individually.  Another 
interesting  finding  was  the  high  number  of  on-task  interactions  (questions,  answers  and 
discussions)  between  members  of  the  dyads,  which  seems  to  support  the  idea  that  a 
computer-mediated  collaborative  environment  promotes  peer-to-peer  communication 
directly  related  to  the  learning  task. 

Computer-Mediated  Collaborative  Group  Size 

Another  variable  that  may  influence  achievement  in  a  computer-mediated 
collaborative  setting  is  the  size  of  the  collaborative  group.  Although  little  empirical 
exploration  exists  on  this  subject  (Collins  &  Onweugbuzie,  2000;Valcke,  1988),  the 
general  rule-of-thumb  is  that  small  collaborative  groups  (2-4)  are  better  than  large  groups 
(Johnson  &  Johnson,  1988).  The  studies  that  have  been  conducted  addressing  this 
question  have  yielded  contradictory  results.  For  example,  Lou,  Abrami  and  d’Apollonia 


(2001)  conducted  a  meta-analysis  of  122  studies  that  compared  small  group  and 
individual  learning  with  computers  and  found  that  groups  of  two  had  significantly  higher 
achievement  than  groups  of  three  to  five  members.  Collins  &  Onweugbuzie  (2000) 
discovered  that  group  size  had  a  quadratic  effect  on  the  quality  of  output  produced  in  a 
face-to-face  undergraduate  research  methodology  class.  This  quadratic  effect  meant  that 
teams  of  two  outperformed  the  teams  of  three,  four  and  five  participants,  but  the  team  of 
six  members  outperformed  all  teams.  But  the  researchers  point  out  that  since  the 
interactions  between  team  members  were  not  recorded,  it  is  unclear  whether  group  size 
was  the  only  factor  that  affected  performance  (Collins  &  Onweugbuzie,  2000). 

However,  Lotan,  Cohen,  and  Hothuis  (1994)  found  somewhat  different  results  in 
a  naturalistic  study  in  which  7th  and  8th  grade  students  were  observed  working  on  ill- 
defined  problems  with  open-ended  solutions  in  social  studies  classes.  In  this  study  no 
groups  were  assigned,  but  the  students  were  allowed  to  form  into  spontaneous  working 
groups.  The  observations  revealed  that  groups  with  the  higher  number  had  the  highest 
level  of  participation  and  the  highest  average  achievement.  The  researchers  concluded 
from  the  observation  data  that  the  larger  the  group  size,  the  larger  the  proportion  of 
students  interacting.  Alternatively,  Mclsaac  and  Ralston  (1996)  observed  a  distance 
education  undergraduate  course  and  found  that  collaborative  groups  larger  than  five  in 
synchronous  environments  may  require  a  moderator  to  be  effective.  However,  it  is 
important  to  point  out  that  the  students  observed  by  Mclsaac  were  involved  in  classroom 
discussions  and  not  working  on  problem-solving  tasks. 
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The  size  of  a  computer-mediated  collaborative  group  in  synchronous 
environments  is  particularly  important  to  consider  when  solving  complex  tasks.  The 
most  commonly  used  form  of  synchronous  computer-mediated  communication  in 
distance  education  settings  is  some  form  of  text-based  communication,  such  as  the 
Internet  Relay  Chat.  An  obvious  advantage  of  this  type  of  system  is  that  participants  can 
communicate  real-time  without  the  delay  experienced  in  asynchronous  systems  like  email 
or  listservs.  A  synchronous  communication  system  is  also  better  suited  for  problem¬ 
solving  tasks  because  it  more  closely  resembles  a  real-world  situation  (Bastiaens  & 
Martens,  2000).  But,  according  to  Murphy  and  Collins  (1997),  one  of  the  biggest 
disadvantages  of  chat-systems  is  that  turn  taking  in  this  synchronous  environment  can  be 
a  problem  because  there  are  no  visual  clues  such  as  body  movements,  eye-contact,  etc.,  to 
indicate  when  someone  wants  to  enter  a  conversation.  In  this  type  of  system,  all  of  the 
users  tend  to  “talk”  simultaneously  and  several  conversations  may  be  occurring  at  the 
same  time,  creating  a  confusing  environment.  Thus,  as  the  number  of  members  of  a 
synchronous  computer-mediated  collaborative  group  increases,  the  positive  effects  of 
collaboration  when  solving  complex  tasks  may  decline.  The  confusion  created  by  a 
larger  number  of  participants  may  have  a  detrimental  effect  on  the  group’s  ability  to  solve 
problems. 

Research  Questions 

This  study  examined  the  effects  of  two  instructional  methods  (case-based  approach 
and  systematic  approach)  and  collaborative  group  size  (teams  of  two  or  four)  on  learner 
performance  in  solving  ill-defined  problems.  The  case-based  approach  provided  a 


learner  with  a  current  state  (problem),  end-state  (goal)  and  a  solution  to  the  problem.  The 
systematic  approach  confronted  the  learner  with  a  problem  but  taught  a  step-by-step 
process  that  could  be  used  to  solve  it.  Data  on  performance  solving  ill-defined  problems, 
attitudes,  and  time  on  task  were  collected  for  all  participants.  Data  were  also  collected 
on  the  nature  of  the  interactions  between  participants  in  both  sizes  of  collaborative 
groups.  The  processes  used  by  participants  to  arrive  at  a  solution  were  also  observed. 
The  research  questions  for  this  study  were: 

1 .  Do  either  instructional  method  (case-based  approach  vs.  systematic  approach)  or 
collaborative  group  size  (dyads  vs.  quads)  have  an  effect  on  learner  performance  in 
solving  ill-defined  problems  in  a  web-based  environment? 

2.  Do  either  instructional  method  or  collaborative  group  size  have  an  effect  on  time 
spent  on  instruction? 

3.  Do  either  instructional  method  or  collaborative  group  size  have  an  effect  on  time 
spent  solving  problems? 

4.  Do  either  instructional  method  or  collaborative  group  size  have  an  effect  on  learner 
attitudes  toward  collaborative  learning,  computer-mediated  collaboration,  web-based 
instructional  programs,  time  available  for  the  program,  and  applicability  of  problem¬ 
solving  skills  learned  to  other  tasks? 

5.  Do  either  instructional  method  or  collaborative  group  size  have  an  effect  on  the  type 
and  amount  of  communication  that  takes  place  between  team  members? 


Method 


Participants 

The  participants  in  this  experiment  were  130  cadets  from  the  United  States  Air 
Force  Academy.  All  of  the  participants  volunteered  to  participate  in  the  study.  There 
were  99  male  participants  (75%)  and  3 1  female  participants  (25%).  The  study  was 
incorporated  into  a  three-hour  segment  of  instruction  of  the  cadets’  first-year  language 
course.  Although  the  participants’  academic  ability  was  not  measured,  they  were 
expected  to  be  high  academic  achievers  since  the  Air  Force  Academy  has  very  high 
entrance  requirements. 

Materials 

The  materials  for  this  study  were  developed  and  incorporated  into  a  web-based 
interface  through  the  Blackboard  course  management  system  (see  Figure  1).  Blackboard 
is  a  course  management  system  that  provides  faculty  members  the  ability  to  develop  and 
deliver  courses  over  the  World  Wide  Web.  The  following  items  were  embedded  within 
the  Blackboard  system  for  this  study:  an  instructional  program,  problem  scenarios  and 
assessment  questions  relating  to  the  problem  scenarios. 

Instructional  Programs.  Two  different  web-based  instructional  programs  were 
developed  for  this  study.  One  program  taught  the  participants  a  systematic  approach  to 
solve  problems.  Another  program  used  a  case-based  approach  to  teach  students  how  to 
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ORIENTATION; 

In  this  program  you  will  first  work  through  a  tutorial  on  problem  solving  Then  yon  will 
apply  what  yon  have  learned  to  solve  several  problem  scenarios.  You  am  required  to 
work  as  a  team  throughout  the  tutorial  and  problem  scenarios.  Your  final  score  will  be 
determined  by  your  individual  performance  on  the  problem  scenarios  and  by  the  level  of 
collaboration  between  team  members .  BE  SURE  TO  COLLABORATE! 

ACADEMIC  SECURITY: 

The  contents  of  this  program  will  remain  under  academic  security  until  directed 
otherwise  by  the  researcher  or  the  instructor.  Do  not  discuss  the  contents  of  this 
program  with  anyone  outside  the  classroom. 

INSTRUCTIONS: 

1)  Click  on  the  button  labeled  "GROUPS"  and  establish  cormaunicaiion  with  your 
team.  Reduce  the  size  of  the  Virtual  classroom*  screen  to  fit  this  window  and  the 
communication  window  on  the  same  screen. 

2)  Click  the  link  below  to  start  the  tutorial 
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Figure  1.  Blackboard  Interface  and  welcome  screen. 
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solve  problems.  Both  programs  taught  students  how  to  solve  decision-making  problems. 
A  description  of  each  instructional  program  is  presented  below. 

Systematic  Approach:  A  web-based  instructional  program  on  a  systematic 
approach  to  problem  solving  was  developed  for  this  study  (see  Appendix  A).  The 
instructional  program  focuses  on  a  four-step  problem-solving  process  derived  from  the 
Air  Force's  "Six-Step  Problem-Solving  Process"  commonly  taught  to  college  juniors 
enrolled  in  the  Air  Force  ROTC  program.  This  modified  approach  was  intended  to 
provide  students  with  a  tool  to  solve  complex  problems.  The  systematic  instructional 
approach  was  also  developed  based  on  principles  for  systematic  problem  solving  outlined 
by  Gagne  (1985)  and  Van  Merrienboer  et  al  (2002).  The  learning  objectives  of  the 
program  were  for  the  student  to: 

1)  Describe  the  problem  environment  and  write  a  problem  statement. 

2)  Categorize  data  relative  to  the  problem. 

3)  Identify  possible  solutions  that  meet  the  problem  criteria. 

4)  Select  the  best  possible  solution. 

The  problem-solving  process  led  students  through  the  following  four  steps: 
problem  definition,  data  gathering,  developing  and  testing  possible  solutions,  and 
selection  of  the  best  possible  solution.  Within  the  program,  an  agent  (animated  cartoon) 
taught  the  steps  of  the  process  and  used  an  example  scenario  to  show  the  learner  how  to 
apply  each  step.  Throughout  this  portion  of  the  instructional  program,  the  learner  was 
asked  to  discuss  the  information  in  each  step  of  the  process  with  the  rest  of  his  or  her 
team.  For  example,  during  the  instruction  for  step  one  the  learner  was  taught  that  it  is 


important  to  identify  the  individuals  involved  in  the  problem.  He  or  she  was  then  asked 
to  discuss  with  his  or  her  team  who  the  individuals  involved  in  the  problem  are  and  to 
write  an  answer  in  a  space  provided.  The  animated  cartoon  then  showed  the  learner  the 
correct  answer. 

The  teaching  of  the  steps  was  followed  by  a  second  practice  scenario  where  the 
student  was  presented  with  the  problem  and  was  asked  to  find  the  best  possible  solution. 
In  total,  the  student  was  exposed  to  two  different  problem  scenarios  in  the  instructional 
program.  The  learner  was  prompted  to  collaborate  with  his  or  her  team  at  several  points 
throughout  the  learning  program. 

Case-Based  Approach:  A  web-based  instructional  program  that  uses  a  case-based 
approach  for  problem  solving  was  also  developed  for  this  study  (see  Appendix  B).  The 
program  was  designed  using  the  principles  outlined  by  Van  Merrienboer  et  al.  (2002)  for 
using  case  studies  to  teach  problem  solving.  The  program  was  designed  as  an  open-ended 
learning  environment  but  with  some  constraints.  The  first  screen  welcomed  the  student 
and  quickly  familiarized  him/her  with  the  web  interface.  The  student  was  then  told  that 
he  or  she  would  be  learning  how  to  solve  complex  problems  by  reading  how  others  have 
solved  similar  problems. 

The  content  of  this  case-based  instructional  program  had  some  similarities  and 
key  differences  with  the  content  of  the  systematic  approach  to  problem  solving  described 
above.  The  same  two  problem  scenarios  were  used  and  an  animated  agent  also  guided 
the  learner.  However,  the  student  was  not  systematically  taught  the  four  steps  of  the 
problem  learning  process.  Instead,  as  Van  Merrienboer  et  al.  (2002)  suggest,  the  learner 
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was  confronted  with  a  given  state  (problem  scenario),  a  desired  state  (goal)  and  a  best 
possible  solution  as  determined  by  the  individual  involved  in  the  case.  For  example,  the 
participant  was  confronted  with  a  problem  scenario  where  a  cadet  is  facing  a  decision¬ 
making  problem  (given  state).  The  agent  “leads”  the  learner  as  he  implements  a  solution 
based  on  the  information  surrounding  the  problem  to  achieve  a  goal  (desired  state).  As 
recommended  by  Van  Merrienboer,  et  al.  (2002),  the  learner  was  asked  several  thought- 
provoking  questions  throughout  the  program,  which  were  intended  to  trigger  deep 
processing  and  to  develop  a  better  mental  model  of  the  problem  environment.  For 
example,  after  the  learner  read  the  problem  scenario  the  learner  was  asked  to  discuss  the 
following  questions  with  his  or  her  team:  “Can  you  think  of  possible  solutions  to  this 
problem?  What  other  information  would  you  need  to  make  a  decision?”  After  the  agent 
developed  possible  solutions,  the  learner  was  asked,  “What  do  you  think  of  my  possible 
solutions?  Do  you  think  my  solutions  address  the  problem?  Why  or  why  not?”  Finally, 
after  the  participants  read  what  happened  when  a  solution  was  implemented,  they  were 
asked  to  discuss  the  following  question,  “Would  you  consider  this  an  acceptable 
outcome?  Why  or  why  not?” 

Once  the  learner  went  through  one  case  study  he  or  she  had  an  opportunity  to 
practice  with  a  different  problem  scenario.  In  the  practice  exercise,  the  learner  was 
presented  with  a  scenario  and  information  related  to  the  scenario.  The  learner  was  asked 
to  collaborate  with  his  or  her  partners  to  identify  the  best  possible  solution  based  on  the 
information  available.  The  feedback  the  learner  received  in  the  program  was  based  on  a 
list  of  “expert”  solutions  for  the  scenario  generated  from  studies  conducted  by  Hedlund, 


Sternberg  and  Psotka  (2002)  for  the  Army  Research  Institute.  In  this  case-based 
approach,  learners  were  given  the  same  two  problem  scenarios  as  learners  in  the 
systematic  approach  and  the  same  amount  of  practice.  The  key  difference  between  the 
two  instructional  programs  was  the  approach  used.  As  previously  noted,  the  participants 
were  asked  to  collaborate  at  several  points  throughout  both  instructional  programs. 

Assessment  Problem  Scenarios.  The  assessment  scenarios  that  were  used  in  the 
instructional  programs  and  for  assessment  were  modified  versions  of  scenarios  developed 
by  Hedlund  et  al.  (2002)  of  the  Army  Research  Institute  (see  Appendix  C).  The 
participants  reached  the  problem  scenario  through  the  screen  shown  on  Figure  2.  The 
problem  scenarios  were  developed  from  realistic  situations  encountered  by  junior  Army 
officers  and  were  modified  by  the  researcher  to  reflect  an  Air  Force  theme.  The 
modifications  included  changing  items  such  as  Army  company  commander  to  Air  Force 
squadron  commander  or  changing  a  tactical  scenario  involving  soldiers  to  tactical 
scenarios  involving  airmen.  There  were  no  substantial  changes  made  to  the  problems 
faced  by  the  young  officer  or  the  circumstances  surrounding  it.  All  of  the  problem 
scenarios  revolved  around  a  military  theme. 

All  treatment  groups  were  assessed  using  the  same  problem  scenarios.  The 
following  is  an  example  of  the  types  of  problem  scenarios  that  were  used: 

“  You  are  a  new  element  leader.  Your  squadron  is  preparing  to  deploy  as  part  of 
a  rapid-response  contingency  TDY.  You  assemble  your  element  and  tell  everyone  to  start 
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Figure  2.  Blackboard  Interface  for  Assessment  Scenarios. 


packing  equipment  in  preparation  for  the  deployment  that  same  night.  When  you  come 
back  to  inspect  their  deployment  preparation,  you  find  that  your  airmen  have  not  packed 
the  equipment  and  are  talking  to  personnel  from  other  elements  who  are  hanging  around 
the  area.  What  should  you  do?  ” 

As  part  of  the  problem  scenario,  the  participants  had  access  to  additional 
information  in  the  form  of  simulated  interviews  with  the  individuals  involved  in  the 
problem  scenario.  The  collaborative  groups  were  able  to  discuss  the  problem  scenario 
and  related  information.  However,  each  member  of  the  group  had  to  answer  each 
assessment  question  individually.  The  question  the  students  had  to  answer  for  each 
problem  scenario  is  listed  below: 

"What  is  the  BEST  POSSIBLE  solution  for  this  problem?  Make  sure  to  FULLY 
EXPLAIN  the  rationale  you  used  to  develop  your  solution. " 

Scoring  Rubric .  A  scoring  rubric  (Appendix  D)  was  used  to  evaluate  the 
participants'  responses  to  the  assessment  scenarios.  Given  the  nature  of  an  ill-defined 
problem  scenario,  there  were  a  variety  of  optimal  solutions  for  each  problem  scenario. 
Therefore,  the  rubric  was  constructed  to  allow  assessment  of  the  validity  of  the  solution 
proposed  by  the  learner,  and  the  quality  of  the  process  the  student  used  to  arrive  at  the 
solution.  The  rubric  was  designed  to  help  evaluate  learner  performance  in  the  following 
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1)  Current  state  or  problem  definition  (Did  the  learner  understand  what  the 
problem  is?) 

2)  End  state  or  goal  (Did  the  learner  understand  what  the  desired  goal  is?) 

3)  Process  (Did  the  learner  follow  a  logical,  well-supported  path  to  achieve  a 
solution?) 

4)  Recommended  solution  (Did  the  solution  produce  the  desired  end-state?  Is  it 
the  best  possible  solution  given  the  learner’s  understanding  of  the  problem?) 

The  participants  were  given  four  problem  scenarios  to  solve,  however,  they  were 
told  to  concentrate  on  solving  the  first  two  scenarios  and  only  do  the  last  two  scenarios  if 
there  was  extra  time  available.  Problems  three  and  four  were  considered  bonus  problems. 
In  order  to  emphasize  the  quality  of  responses  on  the  first  two  scenarios  and  encourage 
collaboration,  they  were  told  that  the  first  two  scenarios  were  worth  8  points,  while  the 
bonus  problems  were  only  worth  one  point  each.  The  maximum  score  possible  for 
problems  one  and  two  was  16  points.  Although  the  participants  collaborated  on  the 
problem  scenarios,  each  participant  provided  individual  responses  to  each  problem.  The 
participants  overall  score  was  determined  by  the  total  number  of  points  earned  for  each 
scenario.  As  a  hypothetical  example,  if  participant  A  was  able  to  complete  all  four 
scenarios  but  the  quality  of  the  responses  was  low,  participant  A  may  have  earned  four 
points  on  scenario  #1,  three  points  on  scenario  #2,  one  point  for  bonus  scenario  #3  and 
zero  points  for  bonus  scenario  #4.  The  total  score  for  participant  A  was  8  points. 
Alternatively,  if  participant  B  completed  only  the  first  two  scenarios  but  the  quality  of  the 
responses  was  high,  he  or  she  may  have  scored  eight  points  for  scenario  #1,  seven  points 


for  scenario  #2  and  zero  points  for  the  bonus  problems.  However,  the  overall  score  of 
participant  B  would  be  higher  than  participant  A’s  with  a  total  of  16  points. 

The  participant  answers  to  the  assessment  scenarios  were  graded  by  the  primary 
researcher  and  another  individual  trained  by  the  researcher  on  the  scoring  rubric  and 
assessment  scenarios.  The  primary  researcher  is  an  Air  Force  major  with  16  years  of 
experience  dealing  with  problems  related  to  technical  troubleshooting,  decision-making, 
and  leadership  and  is  well-qualified  to  assess  the  quality  of  responses  of  the  participants. 
The  additional  rater  holds  a  Bachelor  of  Science  in  business  administration  and  has  15 
years  of  workplace  and  business  experience.  The  inter-rater  reliability  was  established  by 
having  the  primary  rater  and  additional  rater  blind  score  the  problem  scenarios  for  ten 
participants.  The  scores  for  each  assessment  scenario  were  loaded  into  SPSS  for  each  of 
the  ten  participants  and  analyzed  to  determine  the  correlation  between  the  two  raters. 

The  correlation  was  found  to  be  .94.  Once  the  inter-rater  reliability  was  established,  the 
two  raters  scored  an  equal  number  of  participant  responses. 

Attitude  Survey.  A  22-item  survey  was  developed  to  measure  participant  attitudes 
toward  working  with  one  or  three  partners,  as  well  as  their  attitude  toward  the 
instructional  program  and  toward  transfer  of  the  problem  solving  skills  learned  to  other 
tasks  (see  Appendix  E).  The  survey  contained  1 8  Likert-scale  items,  one  multiple  choice 
question  and  three  open-ended  questions.  Respondents  used  a  four-point  Likert  scale 
(0=strongly  disagree,  3=strongly  agree)  to  rate  their  attitude  toward  working  in  a  web 
environment,  working  with  others  when  solving  problems,  and  their  perception  of 
transfer  of  their  problem-solving  skills  to  other  types  of  tasks.  Coefficient  alpha  was 
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computed  as  an  internal  consistency  estimate  of  reliability  for  the  Likert  portion  of  the 
survey  and  found  to  be  .83.  Thel  8  questions  of  the  Likert  portion  of  the  survey  were 
divided  into  six  different  categories  with  three  questions  each.  These  six  categories  were 
intended  to  gauge  participant  attitudes  towards  working  with  others,  collaborating  on¬ 
line,  appeal  of  instructional  materials,  amount  learned  from  instructional  materials,  time 
available  for  the  program  and  applicability  of  information  learned  to  other  settings.  A 
factor  analysis  was  performed  on  these  1 8  questions  to  determine  if  the  survey  data 
revealed  the  same  categories  or  groupings.  This  procedure  yielded  five  factors.  The 
identified  underlying  themes  were  participant  attitudes  toward:  1)  working  with  others,  2) 
collaborating  on-line,  3)  instructional  materials,  4)  time  available  for  the  program,  and  5) 
transfer  of  the  information  learned  to  other  settings.  The  procedure  used  for  factor 
extraction  is  discussed  in  the  design  and  data  analysis  section  below.  Additionally,  the 
participants  were  asked  to  select  their  preference  for  working  on  complex  problems  from 
the  following  four  choices:  alone,  with  one  partner,  with  two  partners,  or  with  three  or 
more  partners.  The  participants  were  also  asked  to  write  down  the  process  they  used  to 
solve  the  problems,  what  they  liked  best  about  the  program,  and  what  they  would  do  to 
improve  the  program. 

Procedures 

In  accordance  with  the  requirements  of  the  Air  Force  Academy  Institutional 
Research  Board,  one  week  prior  to  the  study,  the  primary  researcher  went  into  every  first- 
year  Spanish,  French  and  German  classroom  and  read  a  recruitment  script  (Appendix  G), 
which  explained  the  purpose  of  the  study  in  general  terms,  described  that  the  top- 


performing  teams  would  receive  a  free  dinner,  and  requested  volunteers  for  the  study. 

The  primary  researcher  conducted  the  briefings  in  Civilian  clothes  to  preclude  the 
possibility  of  subtle  coercion  of  the  cadets.  Once  the  cadets  had  made  a  decision,  those 
who  decided  to  participate  were  given  a  copy  of  the  “Subject’s  Bill  of  Rights”  (Appendix 
H)  and  were  asked  to  read  and  sign  an  Informed  Consent  Document  (Appendix  I). 

Immediately  following  the  recruitment  portion  of  the  briefing,  the  cadets  who 
volunteered  for  the  study  were  given  approximately  20  minutes  of  instruction  on  login 
procedures  for  the  Blackboard  system,  how  to  use  the  Blackboard  system  to  collaborate, 
and  general  instructions  on  how  to  navigate  throughout  the  program.  Any  technical 
issues  related  to  access  to  the  Blackboard  system  were  clarified  during  this  briefing. 
Finally,  the  participants  were  given  a  problem  scenario  and  were  asked  to  write-down  the 
process  they  would  use  to  solve  the  problem  (see  Appendix  F).  This  instrument  provided 
information  on  whether  the  participants  had  previously  learned  a  process  for  problem¬ 
solving. 

After  the  recruitment  briefings  were  completed  and  a  list  of  volunteers  was 
generated,  the  participants  were  randomly  assigned  to  one  of  four  treatment  groups.  The 
treatment  groups  were  divided  by  instructional  method  and  collaborative  group  size  as 
follows: 

Group  1 :  Systematic  Approach  -  Dyads 
Group  2:  Systematic  Approach  -  Quads 
Group  3:  Case-Based  Approach  -  Dyads 
Group  4:  Case-Based  Approach  -  Quads 
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The  first  day  of  the  study,  the  participants  were  given  a  card  which  included  their 
username,  password,  and  collaborative  group  number  as  they  came  into  the  laboratory. 
They  were  then  directed  to  pre-arranged  locations  in  the  computer  laboratory,  which 
ensured  that  each  member  of  a  team  was  physically  separated  from  his  or  her  partners. 
This  procedure  was  carried  out  to  prevent  verbal  or  bodily  communications  between  the 
members  of  the  teams,  thus  simulating  a  distance-  education  environment.  At  each 
station,  each  participant  also  had  access  to  a  hard  copy  of  the  on-screen  instructions. 
Figure  3  shows  the  physical  layout  of  the  laboratory  and  some  of  the  participants  at  their 
assigned  stations.  The  participants  were  verbally  instructed  to  sign-on  to  the  Blackboard 
system  using  their  assigned  username  and  password  and  to  navigate  to  the  study’s  web 
page.  Once  everyone  was  properly  logged-in  to  Blackboard,  the  researcher  told  them  to 
follow  the  instructions  on  the  screen  and  informed  them  that  they  had  until  the  end  of  the 
class  period  to  work  on  the  program. 

The  first  screen  of  the  web-based  program  instructed  the  participants  on  how  to 
collaborate  with  their  partners.  The  instructions  reminded  them  that  although  they  were 
answering  the  questions  individually,  they  would  also  have  a  team  score.  The 
instructions  explained  that  a  high  level  of  collaboration  would  result  in  a  higher  group 
score.  At  this  point  the  participants  were  also  reminded  to  concentrate  on  the  first  two 
assessment  scenarios  and  only  work  on  the  last  two  scenarios  if  there  was  time  available. 
The  communication  between  members  of  the  teams  took  place  using  the  virtual 
classroom  feature  of  the  Blackboard  system  (see  Figure  4).  This  feature  allowed  students 
to  chat  with  their  partners  by  entering  a  virtual  classroom  that  the  researcher  set  up  for 


Figure  3.  Study  participants  at  their  assigned  workstations 
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ACADEMIC  SECURITY: 

Tbt  contents  of  ibis  program  will  remain  under  academic  security  until  directed 
otherwise  by  the  researcher  or  the  instructor.  Do  not  discuss  the  contents  of 
this  program  with  anyone  outside  the  classroom. 

INSTRUCTIONS:  .  ^  _ 

1)  Chck  on  the  button  labeled  "DROOPS"  and  establish  communication  with 
your  team  Reduce  the  size  of  the  ’Virtual  classroom"  screen  to  fit  this  window 
end  the  communication  window  on  the  same  screen, 

2)  Click  the  link  below  to  start  the  tutorial 
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Figure  4.  Blackboard  Interface  with  Virtual  Classroom  window. 


each  team  prior  to  the  start  of  the  study.  Each  team  was  assigned  a  different  virtual 
classroom  to  prevent  cross-flow  of  information  between  teams.  There  were  no 
interactions  between  the  participants  and  the  researcher  and/or  instructors  during  the 
study  except  to  remedy  any  technical  difficulties. 

Design  and  Data  Analysis 

This  study  was  a  posttest  only  2  (case-based  approach  vs.  systematic  approach)  x 
2  (collaborative  group  size:  2  vs.  4)  factorial  design.  The  primary  dependent  variable  was 
student  performance  in  solving  ill-defined  problem  scenarios.  Time  on  learning  task, 
time  on  problem-solving  task,  number  of  problems  solved,  and  learner  attitudes  were  also 
analyzed.  An  analysis  of  the  interactions  between  the  members  of  the  collaborative 
teams  was  also  conducted. 

A  2  x  2  analysis  of  variance  (ANOVA)  were  conducted  on  participants' 
performance  on  assessment  scenarios  one  and  two,  time  on  learning  task,  and  time  on 
problem-solving  task  with  treatment  group  and  group  size  as  the  independent  variables. 

A  Wilcoxon  repeated  measures  test  was  also  performed  on  problems  one  and  two  to 
determine  if  performance  was  significantly  different  between  the  two  scenarios. 
Descriptive  statistics  for  the  number  of  participants  who  completed  bonus  problems  three 
and  four  were  also  computed. 

An  internal  consistency  of  reliability  was  calculated  on  the  18  Likert-type  items  of 
the  survey.  Then,  using  the  procedures  outlined  in  Green,  Salkind  &  Akey  (2000),  a 
factor  analysis  was  conducted  to  determine  any  underlying  themes  in  the  survey.  The 
number  of  underlying  factors  was  determined  using  all  items  with  eigenvalues  greater 
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than  one.  The  factors  were  then  rotated  using  the  VARIMAX  method  in  order  to 
interpret  the  results.  This  method  identifies  the  underlying  themes  or  factors  by 
highlighting  the  items  most  highly  correlated  with  each  other. 

The  first  factor  (“working  with  others”)  included  items  one  through  three.  The 
second  factor  (collaborating  on-line)  included  items  four  through  six.  The  third  factor 
(“instructional  materials”)  included  items  seven  through  twelve.  The  fourth  factor  (“time 
available  for  the  program”)  included  items  thirteen  through  fifteen.  And  factor  number 
five  (“transfer  of  the  information  learned  to  other  settings”)  included  question  seventeen 
and  eighteen.  Survey  item  sixteen  (“The  scenarios  in  this  program  were  realistic  and 
applicable  to  a  future  Air  Force  career”)  did  not  correlate  with  any  other  question  and 
was  therefore  not  included  in  any  of  the  five  factors.  An  average  score  for  each  factor 
was  computed  and  separate  2x2  ANOVAs  were  then  conducted  on  each  factor. 

Chi-square  tests  were  conducted  on  the  multiple-option  question: 

When  solving  complex  problems,  I  prefer  to  work 

a.  By  myself 

b.  With  one  partner 

c.  With  two  partners 

d.  With  more  than  two  partners 

A  two-way  contingency  table  analysis  was  used  to  determine  if  a  significant  relationship 
existed  between  treatment  group  and  the  preferences  expressed  by  the  participants. 
Additional  chi-square  tests  were  conducted  by  treatment  group. 


Finally,  the  interactions  between  members  of  the  teams  were  analyzed  using  both 
qualitative  and  quantitative  analysis  techniques.  The  interactions  between  team  members 
were  automatically  recorded  by  the  Blackboard  system.  These  data  were  then 
categorized  by  type  of  interaction.  For  example  an  entry  was  classified  as  a  “question”, 
“answer”,  “discussion”,  “encouragement”  or  “off-task  behavior.”  Previous  researchers 
have  used  similar  categories  in  studies  examining  small  group  interactions  (Klein  & 
Doran,  1999;  Klein  &  Pridemore,  1994;  Uribe,  et  al,  in  press).  A  percentage  for  each 
category  was  computed  and  a  two-way  contingency  table  analysis  was  conducted  on  the 
data  to  evaluate  if  a  significant  relationship  existed  between  type  of  interaction  and 
treatment  group.  Follow-up  pairwise  comparisons  were  conducted  to  evaluate  the 
differences  between  treatment  groups.  The  Holm’s  sequential  Bonferroni  method  was 
used  to  control  for  Type  I  error. 


Results 

Performance  by  Instructional  Method  and  Group  Size 

The  first  two  research  questions  investigated  the  effect  of  instructional  method 
and  group  size  on  performance  in  solving  ill-defined  problems.  Table  1  shows  the  mean 
scores  and  standard  deviations  for  performance  on  assessment  scenarios  one  and  two. 
Due  to  a  problem  with  the  server  hosting  the  Blackboard  program,  some  of  the 
performance  data  were  lost  resulting  in  uneven  cell  sizes.  The  table  reveals  that  on 
problem  one  participants  averaged  3.12  out  of  8  points  (39%),  SD  =  1 .83,  while  the 
average  score  on  problem  two  was  4.04  (51%),  SD  =  1 .82.  A  Wilcoxon  test  conducted 
on  these  data  showed  that  participants  obtained  significantly  higher  scores  on  problem 
two  than  on  problem  one,  z  =  -4.0,  p  <  .001 . 

When  examined  by  group  size,  participants  who  worked  in  dyads  had  an  overall 
average  of  3.45  (43%),  SD  =  2.94  on  problem  one  and  4.12  (52%),  SD  =  1.91  on  problem 
two.  Participants  who  worked  in  groups  of  four  had  an  average  of  2.76  (35%),  SD  =  1.62 
on  problem  one  and  an  average  of  3.96  (50%),  SD  =  1 .74  on  problem  two.  The  data  also 
show  that  participants  who  used  the  systematic  approach  had  an  overall  average  of  3.07 
(38%),  SD  =  1 .83  for  problem  one  and  scored  4.28  (54%),  SD  =  2.08  on  problem  two. 
Participants  who  used  the  case-based  approach  had  an  overall 


Table  1. 

Mean  scores  and  standard  deviations  for  performance  on  problems  1  &  2 


Instructional  Method 


Group  Size 

Systematic 

Case-Based 

Overall 

Prob 

1 

Prob 

2 

Prob 

1 

Prob 

2 

Prob 

1 

Prob 

2 

Dyads 

Mean 

3.31 

4.73 

3.56 

3.65 

3.45 

4.12 

(SD) 

(1.89) 

(2.22) 

(2.02) 

(1.49) 

(1.95) 

(1.91) 

n 

26 

26 

34 

34 

60 

60 

Quads 

Mean 

2.86 

3.86 

2.67 

4.07 

2.76 

3.96 

(SD) 

(1.78) 

(1.88) 

(1.47) 

(1.61) 

(1.62) 

(1.74) 

n 

28 

28 

27 

27 

55 

55 

Overall 

Mean 

3.07 

4.28 

3.16 

3.84 

3.12 

4.04 

(SD) 

(1.83) 

(2.08) 

(1.84) 

(1.55) 

(1.83) 

(1.82) 

n 

54 

54 

61 

61 

115 

115 

Note.  The  maximum  number  of  points  on  each  problem  scenario  was  8 
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average  of  3.16  (40%),  SD  =  1 .84  on  problem  one  and  scored  3.84  (48%),  SD  =  1.55  on 
problem  two. 

A  2  x  2  ANOVA  conducted  on  the  data  for  problem  one  revealed  that  participants 
working  in  dyads  had  a  significantly  higher  performance  score  than  those  working  in 
quads,  F(l,l  1 1)  =  3.9,  p  =  .05,  partial  r\  =  .034.  There  was  not  a  significant  difference 
by  instructional  method,  F(l,l  11)  =  .01, p  =  .93,  partial  q2<  .001,  and  there  was  no 
significant  interaction,  F(l,l 1 1) =  .42,  p  =  .52,  partial  q2  =  .004.  A  2  x  2  ANOVA 
conducted  on  the  data  for  problem  two  revealed  that  there  was  no  significant  difference 
by  instructional  method,  F(l,l  11)  =  .44, p  =  .51 ,  partial  q2=  .004,  or  by  group  size, 

F(  1 , 1 1 1 )  =  1 .65 ,  p  =  .20,  partial  r|2=  .015.  The  data  also  showed  there  was  not  a 
significant  interaction  present,  F(l,l  1 1)  =  3.71,/?  =  .06,  partial  r|2=  .032. 

When  the  scores  for  problems  one  and  two  are  combined,  the  data  reveal  that  the 
overall  scores  for  problems  one  and  two  was  7. 1 7  (45%),  SD  =  2.82.  Participants  who 
worked  in  dyads  scored  an  average  of  7.57  (47%),  SD  =  2.84,  while  participants  who 
worked  in  quads  obtained  an  average  score  of  6.73  (42%),  SD  =  2.75.  The  results  by 
instructional  method  reveal  that  participants  who  used  the  systematic  approach  scored  an 
average  of  7.35  (46%),  SD  =  2.76,  while  participants  who  used  the  case-based  approach 
earned  an  average  of  7. 1 7  (45%),  SD  =  2.88.  ANOVA  conducted  on  these  combined 
scores  did  not  indicate  a  significant  difference  by  group  size,  F(l,l  1 1)  =  2.89,/?  =  .09, 
partial  q  =  .025,  or  by  instructional  method,  F(l,l  11)  =  .59,/?  =  .45,  partial  q  =  .005,  and 
there  was  no  significant  interaction,  F(l,l  1 1)  =  .67,/?  =  .42,  partial  r|2  =  .006. 


Although  performance  on  the  bonus  problems  (problems  three  and  four)  were  not 
included  in  the  analyses  above,  data  were  collected  on  the  number  of  participants  who 
completed  these  problems.  These  data  revealed  that  of  the  54  participants  who  used  the 
systematic  approach,  48  (89%)  completed  problem  three  and  29  (54%)  completed 
problems  three  and  four.  Of  the  61  participants  who  used  the  case-based  approach,  46 
(75%)  completed  problem  three  and  37  (61%)  completed  problems  three  and  four.  The 
data  also  show  that  of  the  60  participants  who  worked  in  dyads,  52  (87%)  completed 
problem  three,  while  41  (79%)  completed  problems  three  and  four.  Of  the  55  participants 
who  worked  in  quads,  42  (76%)  completed  problem  three  and  25  (60%)  completed  both 
bonus  problems. 

Time  spent  on  instruction 

The  next  research  questions  pertained  to  the  effect  of  instructional  method  and 
group  size  on  time  spent  on  instruction.  As  was  the  case  with  the  performance  data, 
intermittent  problems  with  the  server  caused  a  loss  of  some  time  data.  In  addition  to  these 
losses  due  to  technical  problems,  four  participants  whose  times  exceeded  three  standard 
deviations  from  the  mean  were  considered  outliers  and  were  excluded  from  the  time-on- 
task  analyses.  The  losses  and  elimination  of  these  data  resulted  in  different  cell  sizes  for 
the  time-on-task  analysis. 

The  data  in  Table  2  show  that  participants  spent  an  overall  average  of  3 1 .3 
minutes,  SD  =  13.59,  on  the  tutorial.  When  analyzed  by  group  size,  the  data  revealed  that 
participants  who  worked  in  dyads  spent  an  average  of  29.2  minutes,  SD  =  12.8,  while 
participants  who  worked  in  quads  spent  an  average  of  33.7  minutes,  SD  =  14.1.  When 


Table  2. 

Mean  times  and  standard  deviations  for  time  on  the  tutorial 


Group  Size 

Instructional  Method 
Systematic  Case-Based 

Overall 

Dyads 

Mean 

28.7 

29.5 

29.2 

(SD) 

(11.3) 

(14.1) 

(12.8) 

n 

26 

34 

60 

Quads 

Mean 

29.2 

38.2 

33.7 

(SD) 

(11.7) 

(15.0) 

(14.1) 

n 

27 

27 

54 

Overall 

Mean 

28.9 

33.4 

31.3 

(SD) 

(11.4) 

(15.0) 

(13.6) 

n 

53 

61 

114 

examined  by  instructional  method,  the  participants  who  used  the  systematic  approach 
tutorial  spent  an  average  of  28.9  minutes,  SD  =  1 1.4,  while  participants  who  worked 
through  the  case-based  tutorial  spent  an  average  of  33.4  minutes,  SD  =1 5.0.  A  2  x2 
ANOVA  indicated  that  participants  who  worked  through  the  case-based  tutorial  spent 
significantly  more  time-on-task  than  participants  who  worked  through  the  systematic 
approach  tutorial,  F(l,l  10)  =  3.87,/)  =  .05,  partial  rj2  =  ,03.  There  was  not  a  significant 
difference  by  group  size,  F (1,1 10)  =  3.38,/)  =  .07,  partial  q2  =  .03,  and  no  significant 
interaction  was  present,  F(  1,1 10)  =  2.63,  p  =.11,  partial  r|2  =  .02. 

Time  spent  solving  problems 

The  next  research  questions  pertained  to  the  effect  of  instructional  method  and 
group  size  on  time  spent  solving  problems.  Since  the  participants  were  asked  to 
concentrate  on  problem  scenarios  one  and  two,  time  data  were  collected  and  analyzed  for 
these  two  problems.  Server  problems  also  caused  some  of  these  time  data  to  be  lost, 
resulting  in  uneven  cell  sizes. 

Time  spent  on  assessment  scenario  #1.  The  data  in  Table  3  show  that  the 
participants  spent  an  overall  average  of  9.3  minutes,  SD  =  3.0  on  assessment  scenario 
one.  The  data  also  revealed  that  participants  who  worked  in  dyads  spent  an  average  of  9.0 
minutes  on  the  assessment  scenario,  SD  =  3.1,  while  participants  who  worked  in  quads 
spent  an  average  of  9.7  minutes,  SD  =  2.9.  Additionally,  participants  who  used  the 
systematic  approach  to  solve  the  problem  spent  an  average  of  10.2  minutes,  SD  =  3.5, 
while  participants  who  solved  the  problem  using  the  case-based  approach  spent  an 
average  of  8.7  minutes,  SD  =  2.5.  A  2  x  2  ANOVA 


Table  3. 

Mean  times  and  standard  deviations  for  time  spent  on  scenario  #1 


Group  Size 

— 

Instructional  Method 
Systematic  Case-Based 

Overall 

Dyads 

Mean 

9.5 

8.8 

9.0 

(SD) 

(4.2) 

(2.3) 

(3.1) 

n 

17 

30 

47 

Quads 

Mean 

10.9 

8.7 

9.7 

(SD) 

(2.5) 

(2.8) 

(2.9) 

n 

17 

20 

37 

Overall 

Mean 

10.2 

8.7 

9.3 

(SD) 

(3.5) 

(2.5) 

(3.0) 

n 

34 

50 

84 

37 

indicated  that  participants  who  used  the  systematic  approach  spent  significantly  more 
time  on  the  scenario  than  participants  who  used  the  case-based  approach,  F( 1,80)  =  4.86, 
p  =  .03,  partial  x\2=  .06.  There  was  no  significant  difference  by  group  size,  F(l,80)  = 
1.00,/?  =  .32,  partial  r)  =  .012,  and  no  significant  interaction  was  present,  F(l,80)  =  1.22, 
p=  .27,  partial  q2  =  .015.  It  is  important  to  point  out  that  Levene’s  test  for  homogeneity 
of  variance  yielded  a  significant  result  indicating  the  possibility  this  assumption  may 
have  been  violated.  Therefore,  some  caution  should  be  exercised  when  interpreting  these 
data. 

Time  spent  on  assessment  scenario  #2.  The  data  in  Table  4  show  that  the 
participants  spent  an  overall  average  of  9.5  minutes,  SD  =  3.9  on  assessment  scenario 
two.  The  data  also  revealed  that  participants  who  worked  in  dyads  spent  an  average  of  8.5 
minutes  on  the  assessment  scenario,  SD  =  2.6,  while  participants  who  worked  in  quads 
spent  an  average  of  10.6  minutes,  SD  =  3.9.  Additionally,  participants  who  used  the 
systematic  approach  to  solve  the  problem  spent  an  average  of  1 1 .7  minutes,  SD  =  3.6, 
while  participants  who  solved  the  problem  using  the  case-based  approach  spent  an 
average  of  8.0  minutes,  SD  =  2.4. 

A  2  x  2  ANOVA  indicated  that  participants  who  used  the  systematic  approach 
spent  significantly  more  time  on  the  scenario  than  participants  who  used  the  case-based 
approach,  F(l,80)  =  27.9, p  <  .001,  partial  r|2=  .26.  The  data  also  showed  that 
participants  who  worked  in  quads  spent  significantly  more  time  on  the  problem  than 
participants  who  worked  in  dyads,  F(l,80)  =  8.14, p  <  .006,  partial  q2  =  .09.  ANOVA 


TaWc  4. 

Mean  times  and  standard  deviations  for  time  spent  on  scenario  #2 


Group  Size 
Dyads 


Quads 


Instructional  Method 


Systematic 

Case-Based 

Overall 

Mean 

9.6 

7.9 

8.5 

(SD) 

(3.2) 

(2.0) 

(2.6) 

n 

14 

29 

43 

Mean 

13.1 

8.1 

10.6 

(SD) 

(3.1) 

(3.0) 

(3.9) 

n 

21 

20 

41 

Mean 

11.7 

8.0 

9.5 

(SD) 

(3.6) 

(2.4) 

(3.9) 

n 

35 

49 

84 

Overall 


39 


101 


Instructional  Method 


Systematic  Approach 


Case-Based  Approach 


dyads 


quads 


Group  Size 


Figure  5.  Interaction  between  method  and  group  size  for  time  on  problem  #2 
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also  indicated  a  significant  interaction  between  group  size  and  instructional  method, 

F( 1,80)  =  6.83,/?  =  .01,  partial  q2  =  .08  (see  Figure  5).  Follow-up  analyses  were 
conducted  to  evaluate  the  two  simple  main  effects  for  instructional  method  and 
collaborative  group  size.  To  control  for  Type  I  error  across  the  simple  main  effects,  I  set 
alpha  for  each  test  at  .025  (.05/2).  The  results  revealed  a  significant  difference  between 
participants  who  worked  in  dyads  (M=9.6,  SD  =  3.3)  and  those  who  worked  in  quads  (M 
=13.1,  SD  =  3.2)  using  the  systematic  approach,  F(l,  80)  =  12.8,/?  =  .001.  The  analysis 
also  showed  that  for  participants  who  worked  in  quads,  there  was  a  significant  difference 
between  participants  who  used  the  systematic  approach  (M =13 A,  SD  =  3.2)  and  those 
who  used  the  case-based  approach  (M=8.1,  SD  =  3.0),  F(l,  80)  =  32.5, p  <  .001.  The 
analysis  did  not  reveal  a  significant  difference  between  dyads  and  quads  using  the  case- 
based  approach,  F(l,  80)  =  .04,/?  =  .852,  and  there  was  no  significant  difference  between 
dyads  who  used  the  systematic  approach  and  dyads  who  used  the  case-based  approach, 
F(l,  80)  =  3.43,/?  =  .068. 

Participant  Attitudes 

The  next  research  question  investigated  the  effect  of  instructional  method  and 
group  size  on  participant  attitudes.  The  survey  included  18  Likert  items,  one  multiple- 
option  question  and  three  open-ended  questions.  The  18  Likert  items  in  the  survey  were 
rated  by  the  participants  on  a  scale  from  zero  to  three  (0  =  Strongly  Disagree,  1  = 

Disagree,  2  =  Agree,  3  =  Strongly  Agree).  An  internal  consistency  estimate  of  reliability 
for  these  items  was  computed  to  be  .83.  Factor  analysis  of  the  18  survey  items  revealed 
five  factors  or  underlying  themes  in  the  survey  questions.  The  five  identified  factors 
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were  attitudes  toward:  1)  working  with  others,  2)  collaborating  on-line,  3)  instructional 
materials,  4)  time  available  for  the  program,  and  5)  transfer  of  the  information  learned  to 
other  settings.  Item  number  16  in  the  survey  (“The  scenarios  in  this  program  were 
realistic  and  applicable  to  a  future  Air  Force  officer”)  did  not  correlate  with  any  other 
item  in  the  survey  and  it  was  not  included  in  any  of  the  five  factors  mentioned  above. 
However,  it  is  important  to  note  that  this  item  was  the  highest  rated  item  in  the  survey 
with  an  average  rating  of  2.61 . 

An  average  rating  per  factor  was  computed  by  summing  the  scores  for  each  item 
in  the  factor  and  dividing  by  the  number  of  items.  Table  5  shows  the  results  for  each  of 
the  five  factors  for  each  independent  variable.  The  data  reveal  that  participants  rated 
factor  number  one  highest  (“working  with  others”),  M-  2.22,  SD  - . 51  and  factor  two 
(“collaborating  on-line”)  lowest  with  an  overall  average  of  1.68,  SD  =  .76.  A  2  x  2 
ANOVA  of  each  factor  revealed  mixed  results.  Factor  one  (working  with  others)  did  not 
reveal  a  significant  difference  by  method  (F( 1,107)  =  .002,  p  =  .96,  partial  r\2  <  .001), 
group  size  (F(l,107)  =  .29, p  =  .59,  partial  r\=  .003),  and  there  was  no  significant 
interaction  (F( 1 , 1 07)  =  1.06, p  =  .31,  partial  r|2=  .01).  The  data  for  factor  two 
(collaborating  on-line)  did  not  show  a  significant  main  effect  for  group  size  or 
instructional  method,  but  it  did  reveal  a  significant  interaction,  F(l,107)  =  1.02,p-  .009, 
partial  r\2  =  .06  (see  Figure  6).  Follow-up  simple  main  effects  analysis  revealed  a 
significant  difference  between  participants  who  worked  in  dyads  (M =1.44,  SD  =  .76)  and 
those  who  worked  in  quads  (M= 1 .89,  SD  =  .75)  using  the' systematic  approach,  F(  1, 107) 
=  5.1  ,p-  .025.  There  were  no  significant  differences  for  the  other  simple  main 
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Table  5. 

Means  and  standard  deviations  for  each  attitude  survey  category 


Attitude  Factor 

Systematic 
Approach 
(«= 54) 

Case-Based 
Approach 
(«= 57) 

Dyads 

02=56) 

Quads 

02=55) 

Total 

02=111) 

1 .  Working  with  others 

Mean 

SD 

2.22 

.56 

2.22 

.46 

2.24 

.53 

2.19 

.48 

2.22 

.51 

2.  Collaborating  on-line* 

Mean 

SD 

1.67 

.78 

1.68 

.74 

1.64 

.79 

1.72 

.73 

1.68 

.76 

3.  Instructional 

Mean 

1.95 

1.93 

1.96 

1.92 

1.94 

materials 

SD 

.53 

.44 

.56 

.41 

.49 

4.  Time  available  for 

Mean 

2.06 

1.96 

2.13 

1.89 

2.01 

the  program** 

SD 

.58 

.57 

.55 

.58 

.58 

5.  Transfer  of  the  .  _  . 

information  learned  to 

Mean 

2.17 

2.03 

2.05 

2.15 

2.09 

other  settings. 

SD 

.48 

.53 

.58 

.42 

.51 

Note.  Responses  were  on  a  scale  of  0  to  3  where  0  =  Strongly  disagree,  1  =  Disagree,  2  = 
Agree  and  3  =  Strongly  agree.  *  Significant  interaction,/?  =  .009,  **Significant  main 
effect  by  group  size,/?  =  .026 
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Systematic  Approach  Case-Based  Approach 

Instructional  Method 


Figure  6.  Significant  Interaction  for  factor  two  (“collaborating  on-line”) 
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effects  comparisons.  Factor  three  (instructional  materials)  did  not  indicate  a  significant 
difference  by  instructional  method  (F(l,107)  =  .332, p  =  .56,  partial  r\  =  .003),  by  group 
size  (F(l,107)  =  .317, p  =  .58,  partial  r|2=  .003)  or  a  significant  interaction  (F(l,107)  = 
.176,/?  =  .675,  partial  r\  =  .002).  Factor  four  (time  available  for  the  program)  indicated  a 
significant  main  effect  by  group  size,  F(l,107)  =  5.09,  p  =  .026,  partial  r|2=  .05.  There 

2 

was  no  significant  difference  by  instructional  method  (F(l,107)  =1.01  ,p  =  .32,  partial  r\ 

=  .01),  or  a  significant  interaction  (F(l,107)  =  .37, jj  =  .55,  partial  t|2  =  .003).  Finally,  the 
analysis  for  factor  five  (transfer  of  the  information  learned  to  other  settings)  did  not 
indicate  a  significant  difference  by  instructional  method  (F(l,107)  =  2.01  ,p  =  .16,  partial 
r)2=  .018),  by  group  size  (F(  1,107)  =  .99, p  =  .32,  partial  r|2  =  .001),  and  there  was  no 
significant  interaction  (F(l,107)  =  .18,/?  =  .68,  partial  r|2=  .002). 

Participant  Preferences.  The  attitude  survey  also  included  one  option  question 
that  asked  the  participants  to  choose  a  preference  when  solving  complex  problems.  The 
choices  were: 

a.  By  myself 

b.  With  one  partner 

c.  With  two  partners 

d.  With  more  than  two  partners 

Table  6  shows  the  choices  made  the  participants  by  treatment  group.  The  data  are  also 
graphically  depicted  in  Figure  7.  A  two-way  contingency  analysis  of  these  data  indicated 


Table  6. 

Participant  preference  when  working  on  complex  problems  by  treatment  group 


Treatment  Group 

Total 

(%) 

Preference 

System 

Dyads 

System 

Quads 

Case-Based 

Dyads 

Case-Based 

Quads 

Alone 

3 

3 

4 

2 

12 

(11%) 

With  one  partner 

11 

9 

12 

7 

39 

(35%) 

With  two 

8 

9 

9 

10 

36 

partners 

(33%) 

With  more  than 

3 

7 

5 

8 

23 

two  partners 

(21%) 

Total 

(%) 

25 

(23%) 

28 

(25%) 

30 

(27%) 

27 

(25%) 

110 

(100%) 

Number  of  participants 


Figure  7.  A  clustered  bar  chart  of  preference  for  working  on  complex  problems  by 
treatment  group. 


a  non-significant  relationship  between  treatment  group  and  preference,  %2(9,  N=1 10)  = 
4.6,  p  —  .87,  Cramer ’s  V—  0.12.  When  the  preference  data  is  analyzed  for  all  participants 
regardless  of  their  treatment  group,  the  results  show  that  of  die  1 10  participants  who 
answered  this  question,  12  (1 1%)  indicated  a  preference  to  work  alone,  39  (35%) 
indicated  a  preference  to  work  with  one  partner,  36  (33%)  prefer  to  work  with  two 
partners,  and  23  (21%)  indicated  they  prefer  to  work  with  more  than  two  partners  when 
solving  complex  problems,  %2(3,  N=1 10)  =  17.6,  e  =  .001,  effect  size  =  +0.16. 

Problem-solving  process  used  by  the  participants.  Prior  to  the  treatment  the 
participants  were  given  a  simple  problem  scenario  and  were  asked  to  state  the  process 
they  would  use  to  solve  it.  The  responses  indicated  that  91%  (1 1 8)  of  the  participants  did 
not  know  a  process  or  heuristic  to  solve  ill-defined  problems.  Some  of  the  answers 
included: 

•  “None" 

•  “I  would  have  a  talk  with  the  player  ” 

•  “I  would  first  attempt  to  talk  to  the  team  ” 

After  the  treatment,  the  participants  were  asked  to  state  the  process  they  used  to  solve  the 
problem  scenarios.  When  the  responses  to  this  question  are  analyzed  by  instructional 
method,  the  data  show  that  5 1  out  of  60  participants  who  learned  the  systematic  approach 
indicated  they  used  the  approach  they  learned  in  the  tutorial  to  solve  the  problems.  Some 
examples  of  the  responses  are: 
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•  “  1)  Evaluated  the  problem;  2)  acknowledged  who  was  involved;  3)  determined 
any  constraints;  4)  brainstormed  until  a  good  solution  came  to  mind  that  fit  with 
in  the  constraints.” 

•  “We  determined  the  people  involved,  determined  the  goal,  then  found  solutions, 
taking  into  consideration  any  obstacles.  ” 

•  “Determine  the  people  involved.  Determine  the  problem.  Determine  any 
obstacles.  Prepare  a  possible  solution  ” 

Of  the  60  participants  in  the  case-based  approach  who  responded  to  this  question,  39 
indicated  they  used  a  step-by-step  process  to  solve  the  problems.  Some  of  these 
responses  included: 

•  “Find  the  problem  -ask  question  and  become  as  knowledge  [sic]  as  possible  on 
the  subject  and  relative  things,  -come  up  with  possible  solutions  -choose  the  best 
solution  or  solutions.  ” 

•  “Identified  the  problem,  evaluated  the  possible  solutions,  then  chose  the  best 
one.  ” 

•  “ Figure  out  problem— discuss  problem  with  partner— come  up  with  new  ideas 
based  on  partner's  responses— answer  problem.  ” 

•  “Read  the  problems.  2.  Came  up  with  solutions  on  my  own.  3.  Discussed 
possibilities  with  team.  4.  Picked  the  best  solution.  ” 

Open-ended  questions  about  the  program.  Table  7  shows  the  top  three  responses 
by  instructional  method  to  the  open-ended  questions  “What  did  you  like  best  about  the 
program?”  and  “What  would  you  do  to 
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Table  7. 

Responses  to  attitude  survey  open-ended  questions  by  instructional  method 


Instructional  Method 


Question 

Systematic  Approach 

Case-Based  Approach 

What  did  you  like  best 
about  the  program? 

•  The  realism  of  the 
problem  scenarios. 

(n  =  24) 

•  The  instructional 
materials. 

(n  —  21) 

•  Working  with  others. 

(n=  13) 

•  Working  with  others. 

(n  =  22) 

•  The  realism  of  the 
problem  scenarios. 

(n=  18) 

•  The  instructional 
materials. 

(n  =  11) 

What  would  you  do  to 

•  Improve  the  tutorial  (i.e. 

•  Improve  the  tutorial  (i.e. 

improve  the  program? 

make  the  assessment 

make  the  assessment 

scenarios  more 

scenarios  more 

complex,  improve  the 

complex,  improve  the 

navigation,  more 

navigation,  more 

practice,  etc.) 

practice,  etc.) 

(n=21) 

(n=20) 

•  Allow  more  time  to 

•  Allow  more  time  to 

complete  the  program. 

complete  the  program. 

(n  =  8). 

(n  =  12). 

•  Use  a  different  medium 

•  Use  a  different  medium 

for  communication 

for  communication 

between  team  members. 

between  team  members. 

(n  =  4) 

(n=7). 
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improve  the  program?”  The  data  show  that  the  top  response  for  what  participants  liked 
best  in  the  systematic  approach  tutorial  was  the  realism  of  the  assessment  scenarios, 
while  the  participants  in  the  case-based  approach  tutorial  felt  working  with  others  was  the 
best  thing  about  the  program.  The  table  also  shows  that  participants  in  both  instructional 
methods  felt  changes  to  the  instructional  program  were  the  best  way  to  improve  the 
program.  These  changes  included  making  the  assessment  scenarios  more  complex, 
adding  additional  practice  and  feedback,  and  improving  the  navigation  throughout  the 
tutorial. 

Learner-to-Learner  Interactions 

The  last  research  question  dealt  with  the  effect  of  instructional  method  and  group 
size  on  the  type  and  quantity  of  communications  between  team  members.  The  data  in 
Table  8  show  that  there  were  a  total  of  3,665  communications  between  members  of  the 
groups.  These  communications  were  categorized  into  one  of  five  possible  categories:  1) 
questions,  2)  answers,  3)  discussions,  4)  encouragement,  and  5)  off-task.  The  data  show 
that  17%  were  "questions",  16%  were  "answers",  59%  were  "discussions",  3%  were 
“encouragement”  and  5%  were  "off-task"  entries. 

A  two-way  contingency  analysis  of  these  data  revealed  that  treatment  group  and 
communication  category  were  significantly  related,  y2(12,  N=3665)  =  62.8,  g  <  .001, 
Cramer’s  V=  0.08.  Follow-up  pairwise  comparisons  were  conducted  to  evaluate  the 
differences  among  the  four  treatment  groups.  The  Holm’s  sequential  Bonferroni  method 
(Green,  et  al.,  2000)  was  used  to  control  for  Type  I  error  at  the  .05  level  across  all  six 
comparisons.  The  data  in  Table  9  shows  that  three  comparisons  (system-dyads  vs.  case- 


Table  8. 

Participant  communications  by  treatment  group  and  interaction  category 


Treatment  Group 

Communication 

Category 

System 

Dyads 

System 

Quads 

Case-Based 

Dyads 

Case-Based 

Quads 

Total 

(%) 

Question 

163 

142 

184 

118 

607 

(17%) 

Answer 

153 

129 

179 

108 

569 

(16%) 

Discussion 

558 

528 

724 

370 

2180 

(59%) 

Encouragement 

23 

26 

79 

9 

137 

(3%) 

Off-Task 

64 

45 

35 

28 

172 

(5%) 

Total 

(%) 

961 

(26%) 

870 

(24%) 

1201 

(33%) 

633 

(17%) 

3665 

(100%) 

dyads,  system-quads  vs.  case-dyads,  and  case-quads  vs.  case-dyads)  of  the  total  number 
of  communications  were  found  to  be  significant. 


Table  9. 

Results  for  the  pairwise  comparisons  using  the  Holm ’s  sequential  Bonferroni  method 


Comparison 

Pearson 

Chi- 

Square 

p- value 

Required  p- 
value  for 
significance 

Significance 

Cramer’s  V 

Case-Dyads  vs. 
Case-Quads 

29.9 

<.001 

.008 

* 

.13 

System-Dyads  vs. 
Case-Dyads 

37.9 

<.001 

.01 

* 

.13 

System-Quads  vs. 
Case-Dyads 

19.8 

.001 

.0125 

* 

.10 

System-Quads  vs. 
Case-Quads 

6.9 

.14 

.0167 

NS 

.07 

System-Dyads  vs. 
Case-Quads 

6.0 

.197 

.025 

NS 

.06 

System-Dyads  vs. 
System-Quads 

3.3 

.51 

.05 

NS 

.04 

Note.  *  Significant  Comparison 


Discussion 

Performance  by  Instructional  Method  and  Group  Size 

The  purpose  of  this  study  was  to  investigate  the  effects  of  instructional  method 
and  computer  mediated  collaborative  group  size  on  learner  performance  in  solving  ill- 
defined  problems.  Overall,  participants  performed  significantly  better  on  problem  two 
than  on  problem  one.  There  are  two  possible  reasons  for  this  result.  First,  it  is  likely  that 
problem  one  provided  the  participants  additional  practice,  which  improved  their 
performance  on  problem  two.  It  is  also  possible  that  problem  two  was  easier  to  solve 
than  problem  one.  Although  both  problems  dealt  with  junior  officers  facing  a  decision¬ 
making  dilemma,  the  characteristics  of  problem  one  made  could  have  made  it  more 
difficult  for  the  participants  to  solve. 

Separate  analyses  of  the  problems  indicated  that  participants  who  worked  in 
computer-mediated  collaborative  dyads  performed  significantly  better  than  did 
participants  who  worked  in  computer-mediated  collaborative  quads.  This  finding 
supports  previous  research  that  showed  dyads  perform  better  than  larger  groups  ( Johnson 
&  Johnson,  1988;  Lou,  et  al.,  2001;  Collins  &  Onweugbuzie,  2000).  However,  in  this 
study  dyads  performed  significantly  better  only  on  problem  one.  The  fact  dyads 
interacted  more  than  quads  throughout  the  study  may  have  contribured  to  this  result. 
Although  all  participants  were  able  to  discuss  the  problem  and  related  information  with 
others,  the  results  seem  to  suggest  that  participants  who  worked  in  dyads  extracted  more 


benefit  from  these  leamer-to-leamer  interactions  than  participants  who  worked  in  quads, 
which  positively  affected  their  performance  on  the  more  complex  task. 

The  finding  that  a  higher  level  of  collaboration  results  in  better  performance  is 
supported  by  findings  from  other  studies  where  participants  worked  in  a  collaborative 
learning  environment  (Chang  &  Smith,  1991;  Flynn  &  Klein,  2001;  Johnson  &  Chung, 
1999;  Johnson  et  al.,  1991;  Mergendoller  eta!.,  1999;  Uribe,  et  al.,  in  press).  The  amount 
of  communication  between  members  of  dyads  when  compared  to  those  of  quads  seems  to 
support  the  hypothesis  that  more  communication  between  team  members  resulted  in 
better  performance  on  problem  one.  The  analysis  of  the  team  interactions  showed  that 
dyads  had  a  higher  number  of  communications  than  participants  who  worked  in  teams  of 
four.  This  higher  number  of  communications  may  have  led  to  the  generation  of  a  higher 
number  of  possible  solutions,  and  ultimately  to  better  performance  on  problem  one. 

The  quality  of  leamer-to-leamer  interaction  may  have  also  led  to  better 
performance  by  participants  who  worked  in  dyads  over  those  who  worked  in  quads.  An 
informal  qualitative  analysis  of  the  interaction  data  conducted  by  the  researcher  seemed 
to  indicate  that  participants  who  worked  in  dyads  had  higher-quality  interactions  than 
those  who  worked  in  quads.  The  exchange  of  ideas  and  information  between  members  of 
dyads  appeared  to  probe  deeper  into  the  problem,  thus  generating  better  possible 
solutions;  whereas  the  interactions  between  members  of  quads  seemed  to  be  superficial, 
lacking  depth  and  insightfulness.  The  examples  below  show  a  sample  interaction  for  a 
quad  and  a  dyad  when  solving  problem  scenario  one.  These  examples  are  representative 
of  the  interactions  observed  during  the  study. 


■  56 

Quad  example: 

Student  1  >  we  are  doing  #1  right? 

Student  2  >  yeah 
Student  3  >  right 
Student  4  >  si 

Student  1  >  good,  i  [sic]  am  ready 
Student  4  >  me  too 
Student  2  >  let's  do  it  then 
Student  2  >  what's  our  solution? 

Student  2  >  take  away  the  other  flight  and  remind  the  flight  that  we  have 
to  get  on  board? 

Student  3  >  Bust  there  [sic]  chops. 

Student  1  >  make  them  pack, 

Student  1  >  that  too, 

Student  4  >  make  them  pack 
Student  1  >  get  rid  of  the  other  guys 

Dyad  Example: 

Student  1  >  ok,  I'm  starting  on  the  first  one 
Student  2  >  ME  TOO 
Student  1  >  what  do  you  think? 

Student  2  >  i  [sic]  think  that  b-flight  needs  to  get  out  of  there 

Student  2  >  but,  in  the  scenario,  we're  only  an  [sic]  It  and  the  b-flight  com 

is  a 

captain 

Student  2  >  so  that  could  pose  a  problem 
Student  1  >  good  point 

Student  1  >  but  I  think  the  mission  is  more  important  so  we  should  have 
them  pack  and  then  they  can  chat  with  bflight 
Student  2  >  i  agree 
Student  1  >  is  that  our  solution  then? 

Student  2  >  well,  how  do  we  get  them  to  pack 
Student  1  >  we  order  them  to 

Student  2  >  right,  but  they  will  still  be  distracted  by  b  flight 

Student  2  >  so,  i  think  we  should  order  our  troops  to  pack  (direct  order) 

Student  1  >  we  need  an  incentive  to  pack 

Student  2  >  and  make  it  very  clear  to  the  captain  that  we  want  his  flight 
out  of  there  until  we're  done 
Student  1  >  ok 
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While  group  size  had  a  significant  effect  on  performance,  instructional  method 
did  not.  There  are  two  possible  reasons  for  this  result.  First,  the  participants  who  worked 
through  the  case-based  approach  appeared  to  have  acquired  a  heuristic  that  they  were 
able  to  apply  to  the  problem  scenarios.  The  results  show  that  participants  in  both 
instructional  methods  used  a  process  to  solve  the  problems.  Most  participants  in  the 
systematic  approach  group  indicated  they  used  the  4-step  process  they  learned  in  the 
tutorial,  while  about  65%  of  the  participants  in  the  case-based  approach  used  a  three-step 
heuristic  similar  to  the  one  learned  by  the  participants  in  the  systematic  approach.  This 
seems  to  support  the  notion  that  when  learners  are  faced  with  a  problem  situation,  they 
develop  a  mental  model  that  can  then  be  applied  to  similar  situations  (V anMerrienboer,  et 
al.,  2002)  The  fact  that  both  treatment  groups  applied  a  step-by-step  approach  to  solve  the 
problems  may  have  equalized  their  performance  on  the  assessment  scenarios. 

Another  reason  for  the  non-significant  finding  may  have  been  that  the  time 
limitations  imposed  on  the  study  weakened  the  instructional  programs.  The  survey 
results  show  that  students  felt  they  could  have  benefited  from  additional  practice, 
additional  examples  and,  perhaps  most  importantly,  additional  time.  It  is  important  to 
point  out  that  the  original  instructional  programs  were  shortened  after  two  formative 
evaluations  showed  both  programs  would  take  longer  than  the  time  available  for  this 
study. 

These  reductions  to  the  instructional  programs  may  also  have  led  to  the  overall 
poor  performance  by  both  treatment  groups  on  the  problem  scenarios.  Participants  spent 
an  overall  average  of  3 1.3  minutes  on  the  tutorial,  9.3  minutes  on  problem  one  and  9.5 
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minutes  on  problem  two.  Although  poor  performance  on  problem-solving  tasks  has  been 
evident  in  similar  studies  (Flynn  &  Klein,  2001;  Uribe,  etal,  in  press),  the  short  amount 
of  time  spent  on  instruction  and  a  reduced  set  of  practice  scenarios  may  have  contributed 
to  lower  performance  scores  in  this  study. 

Time  Spent  on  Instruction 

The  results  for  time  spent  on  instruction  revealed  that  participants  who  used  the 
case-based  tutorial  spent  significantly  more  time  on  instruction  than  participants  who 
used  the  systematic  approach.  Case-based  participants  may  have  spent  more  time  on  the 
learning  task  because  of  key  differences  in  the  two  instructional  programs.  In  the 
systematic  approach  participants  were  taught  each  step  of  a  problem-solving  process,  and 
then  were  directed  to  use  each  step  in  a  practice  scenario.  Conversely,  the  case-based 
approach  program  was  designed  to  allow  the  participants  more  opportunity  to  reflect  on 
the  problem  scenarios  and  to  formulate  alternative  paths  to  a  best  possible  solution.  This 
difference  in  the  teaching/leaming  approach  may  have  had  an  impact  on  the  time  spent  on 
the  tutorial. 

Another  explanation  may  be  that  the  participants  in  the  case-based  approach  spent 
more  time  on  instruction  because  of  the  increased  cognitive  load  of  developing  a  heuristic 
or  mental  model  that  could  be  applied  to  other  similar  problems  (Van  Merrienboer, 

2002).  Participants  who  learned  the  systematic  approach  learned  a  heuristic  directly  from 
the  instruction,  and  therefore  did  not  have  the  additional  mental  burden  of  formulating  a 
step-by-step  approach. 
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Time  spent  solving  problems 

In  contrast  to  the  results  for  time  spent  on  instruction,  participants  who  used  the 
systematic  approach  spent  more  time  solving  the  problems  than  did  participants  in  the 
case-based  approach.  Participants  who  used  the  systematic  approach  may  have  spent 
more  time  methodically  applying  the  steps  of  the  problem-solving  process  to  solve  the 
scenarios.  Although  participants  in  the  case-based  approach  also  used  a  heuristic  to  solve 
the  problems,  most  appeared  to  have  used  a  two  or  three-step  process  rather  than  the  full 
four-step  process  used  by  the  systematic-approach  participants.  This  may  have 
contributed  to  the  difference  in  the  time  spent  solving  problems. 

However,  the  results  for  time  spent  solving  problem  two  were  different  than  the 
results  for  problem  one.  In  addition  to  the  significant  difference  between  participants  in 
the  systematic  approach  and  the  case-based  approach  for  time  spent  on  the  problem, 
problem  two  exhibited  a  significant  interaction  between  instructional  method  and  group 
size.  The  data  show  that  quads  who  used  the  systematic  approach  spent  significantly 
more  time  on  the  problem  than  did  quads  using  the  case-based  approach.  However,  there 
was  no  significant  difference  between  dyads  who  used  the  systematic  approach  and  those 
who  used  the  case-based  approach.  Quads  who  used  the  systematic  approach  may  have 
spent  more  time  on  the  problem  because  applying  a  more  structured  process  to  solve  the 
problems  required  additional  coordination  than  applying  a  more  open-ended  case-based 
approach. 

In  addition  to  the  higher  degree  of  coordination  required  to  apply  a  systematic 
approach  to  solve  problems,  confusion  arising  from  synchronous  computer-mediated 
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communication  in  large  groups  might  have  also  contributed  to  these  results.  An  informal 

qualitative  analysis  of  the  data  indicates  that  participants  who  worked  in  quads 

experienced  some  confusion  during  the  communication  process.  The  larger  number  of 

team  members  in  the  quads  might  have  been  a  factor  in  creating  a  confusing  learning 

environment.  This  is  in  line  with  other  research  that  has  shown  larger  groups  may  require 

a  moderator  to  be  effective  (Mclsaac  and  Ralston,  1996).  The  attempt  to  apply  a  more 

structured  approach  in  this  somewhat  disorganized  communication  environment  may 

have  led  quads  to  spend  more  time  solving  problem  two  than  participants  working  in 

dyads.  The  example  below  shows  a  typical  exchange  between  four  members  of  a  quad 

where  the  conversation  seems  somewhat  disjointed: 

Student  1  >  hey  guys 
Student  2  >  yeah 
Student  2  >  sounds  good 
Student  1  >  I  just  read  that  thingy 
Student  3  >  what  thingy 

Student  2  >  are  we  "ready  to  answer  the  questions"? 

Student  1>  sure 
Student  4  >  yes 

Student  3  >  you  guys  done  with  that 
Student  2  >  with  what? 

Student  3  >  with  those  answers 
Student  2  >  not  yet 
Student  4  >  yes 


The  data  for  number  of  participants  who  completed  bonus  problems  also  appears 
to  indicate  quads  may  not  be  an  ideal  grouping  when  working  on  complex  problems.  The 
results  show  that  participants  who  worked  in  quads  in  both  instructional  methods  were 
less  likely  to  complete  both  bonus  problems  than  were  participants  who  worked  in  dyads. 


Participant  attitudes 

hi  general,  participants  indicated  a  preference  for  working  collaboratively.  The 
data  specifically  show  that  a  majority  of  participants  prefer  to  work  with  one  or  two 
partners  when  solving  ill-defined  problems.  Other  research  has  also  shown  a  preference 
by  learners  to  work  in  collaborative  environments  (Flynn  &  Klein,  2001 ;  Johnson  et  al., 
1991;  Lou  et  al.,  2001,  Uribe,  et  al.,  in  press).  However,  collaborating  on-line  was  rated 
lowest  (1.68  out  3.0)  by  the  participants  indicating  that  learners  did  not  necessarily  enjoy 
collaborating  using  a  text-based  system.  This  supports  other  research,  which  found  that 
participants  did  not  enjoy  on-line  collaboration  due  to  the  difficulties  of  communicating 
via  a  computer  (Uribe,  et  al.,  in  press). 

However,  participant  attitudes  toward  collaborating  on-line  were  not  constant 
across  the  instructional  methods.  Participants  who  worked  in  quads  and  used  the 
systematic  approach  liked  collaborating  on-line  significantly  more  than  dyads  who  used 
the  systematic  approach;  whereas,  there  was  not  a  significant  difference  between  dyads 
and  quads  who  used  the  case-based  approach.  Since  dyads  appear  to  have  attempted 
more  in-depth  discussions  of  the  problems,  they  may  have  felt  more  frustrated  attempting 
to  collaborate  in  a  more  structured  environment.  However,  participants  who  worked  in 
quads  did  not  experience  the  same  level  of  frustration  using  the  systematic  approach 
because  their  communications  remained  at  a  superficial  level. 

The  survey  results  also  show  that  the  participants  enjoyed  the  Internet-based 
learning  materials.  The  instructional  materials  were  listed  as  one  of  the  top  three  things 
participants  liked  best  in  both  instructional  methods.  This  finding  supports  other  research 


62 

where  student  attitudes  toward  web-based  instruction  have  been  found  to  be  positive 
(Adelskold,  1999;  Mclsaac  &  Ralston,  1996;  Savenye,  2001;  Uribe,  et  al.,  in  press).  The 
realism  of  the  problem  scenarios,  cited  by  a  large  number  of  participants  as  something 
they  liked  best  about  the  entire  program,  could  have  also  contributed  to  participant 
enjoyment  of  the  learning  materials. 

The  attitude  survey  also  attempted  to  gauge  the  learner  perceptions  on  the 
applicability  of  the  problem-solving  skills  learned  to  other  settings.  The  results  indicated 
that  most  participants  agreed  that  what  they  had  learned  in  this  study  would  transfer  to 
other  situations.  This  is  in  line  with  similar  studies  that  have  found  a  positive  attitude 
towards  transfer  of  problem-solving  skills  to  similar  situations  (Uribe,  et  al.,  in  press). 
Participants  felt  they  would  use  this  process  in  their  professional  lives  because  the 
problems  they  faced  in  the  study  were  directly  related  to  their  future  career  as  Air  Force 
officers.  Additional  evidence  that  supports  this  hypothesis  is  that  the  highest  rated  survey 
item  dealt  with  the  realism  and  applicability  of  the  scenarios  to  an  Air  Force  career. 
Learner -to-Learner  Interactions 

The  analysis  of  the  leamer-to-leamer  interactions  showed  that  the  number  of 
communications  in  each  category  varied  significantly  as  a  function  of  treatment  group. 
Results  revealed  that  participants  who  worked  in  case-based  dyads  had  significantly  more 
communications  than  the  other  treatment  groups.  These  data  also  suggested  that  dyads 
collaborated  more  than  quads  in  both  the  systematic  and  case-based  approaches.  This 
additional  collaboration  might  have  been  a  factor  in  dyads  performing  better  than  quads 
on  problem  one.  This  finding  is  in  line  with  the  results  found  by  Uribe,  et  al.  (in  press), 
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where  increased  collaboration  directly  contributed  to  better  performance  when  solving  ill- 
defined  problems. 

The  majority  of  communications  that  took  place  between  team  members  were 
related  to  the  task.  These  results  also  show  that  over  half  of  the  communications  were 
categorized  as  “discussion,”  indicating  that  the  participants  in  all  treatment  groups  were 
actively  engaged  in  discussion  of  the  problem  or  related  information.  This  finding 
supports  other  research  where  a  collaborative  learning  environment  produced  a  high 
percentage  of  on-task  interactions  (Flynn  &  Klein,  2001;  Klein  &  Doran,  1999;  Sherman 
&  Klein,  1995;  Uribe,  et  al.,  in  press). 

Implications 

This  study  has  clear  implications  for  the  design  and  delivery  of  distance  learning. 
The  study  indicates  that  grouping  learners  into  dyads  instead  of  quads  maybe  a  better 
strategy  when  learning  and  applying  problem-solving  skills  in  a  web-based  environment. 
When  time-on-task  data  and  number  of  problems  completed  is  considered  in  conjunction 
with  performance  and  group  interaction  data,  findings  seem  to  suggest  that  dyads  are  a 
more  efficient  grouping  than  quads.  The  results  show  that  dyads  communicated  more, 
spent  less  time-on-task,  completed  more  problems  and  performed  better  than  quads  on 
one  of  the  problem  scenarios. 

The  study  also  suggests  that  a  computer-mediated  environment  is  conducive  to 
collaboration.  The  high  percentage  of  on-task  interactions  indicates  that  computer- 
mediated  collaborative  learning  should  be  considered  when  high-interactivity  and 
exchange  of  ideas  is  a  major  component  of  a  distance  learning  program.  But  while 
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learners  appear  to  enjoy  working  together  when  solving  complex  problems,  instructional 
designers  and  instructors  should  keep  in  mind  that  collaborating  using  text-based 
communications  appears  to  have  some  drawbacks.  Additional  time  should  be  factored 
into  any  program  to  allow  the  learner  ample  time  for  communication. 

The  findings  of  this  study  also  indicate  that  the  systematic  approach  and  the  case- 
based  approach  may  be  equally  effective  for  solving  ill-defined  problems.  Some  of  the 
evidence  indicates  that  the  case-based  method  may  be  a  better  approach  if  a  goal  of  the 
learning  program  is  for  the  learner  to  develop  his  or  her  own  rules-of-thumb  or  heuristics 
for  problem  solving.  But  regardless  of  method  used,  the  findings  clearly  suggest  that 
learners  enjoy  scenarios  that  closely  match  the  “real  world.”  If  possible,  problem 
scenarios  representative  of  the  learner’s  future  profession  should  be  used  when  teaching 
how  to  solve  ill-defined  problems. 

Further  Research 

The  results  of  this  study  suggest  other  areas  that  could  be  addressed  by  additional 
research.  The  likely  interdependence  of  the  dependent  measures  in  a  collaborative 
environment  should  be  explored.  Alternative  statistical  analysis  techniques,  such  as  the 
Hierarchical  Linear  Model,  could  be  used  to  avoid  the  potential  pitfalls  of  violating  the 
independence  of  scores  assumption  of  ANOVA. 

Different  computer-mediated  collaborative  group  sizes  should  be  explored  to 
determine  an  optimal  size.  Although  this  study  indicates  that  dyads  may  be  more 
effective  than  quads,  additional  research  should  investigate  alternative  group  sizes.  This 
study  seems  to  indicate  that  as  group  size  increases,  the  efficiency  of  the  group  decreases. 
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Research  into  other  groupings  could  yield  data  that  could  be  used  to  confirm  this  finding 
and  to  develop  a  prediction  model  for  the  effect  of  group  size  on  performance  when 
solving  ill-defined  problems. 

The  instructional  methods  used  to  teach  problem  solving  could  also  benefit  from 
additional  research.  Variations  of  the  systematic  and  case-based  strategies  for  problem 
solving  should  be  explored  to  determine  the  best  approach  for  specific  problem-solving 
tasks.  It  would  be  useful  for  instructional  designers  and  practitioners  to  know  if  a 
particular  approach  is  better  suited  for  certain  types  of  tasks  in  a  computer-mediated 
collaborative  environment.  For  example,  the  ill-defined  problems  identified  by  Jonassen 
(2000)  such  as  decision-making,  strategic  performance,  case  analysis  and  design 
problems  could  be  examined.  Also,  specific  characteristics  of  each  approach,  such  as 
number  of  steps  in  a  systematic  approach  or  types  of  thought-provoking  questions  in  the 
case-based  approach,  could  be  manipulated  to  determine  their  effects  on  performance. 
Research  on  the  different  facets  of  computer-mediated  collaboration  should  help  us 
identify  the  most  effective  instructional  practices  to  effectively  promote  the  learning  of 
problem-solving  skills  in  a  web-based  environment. 
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decision,  it  will  be  very  hard  for  you  to  gain  the  alternative  solution/ 
support  of  the  NCOs  and  troops  to  carry  out  the 
battery  commander's  wishes.  “ 


APPENDIX  E 
PARTICIPANT  SURVEY 
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PARTICIPANT  SURVEY 

We  are  conducting  this  survey  to  get  your  perceptions  about  the  Problem  Solving 
program  you  just  completed.  Your  honest  feedback  will  help  us  determine  areas  for 
improvement.  Please  rate  statements  1-1 8  below  using  the  following  rating  scale: 


SD  -  Strongly  Disagree  D- Disagree  A  -  Agree  SA  -  Strongly  Agree 


1 . 1  enjoyed  working  with  other  people  on  the  program. 

SD 

D 

A 

SA 

2.  The  program  was  easier  to  do  because  I  worked  with 
other  people. 

SD 

D 

A 

SA 

3 .  Working  with  others  helped  me  do  better  on  the 
program. 

SD 

D 

A 

SA 

4. 1  liked  communicating  with  my  team  using  the 
computer. 

SD 

D 

A 

SA 

5.  Collaborating  with  my  team  using  the  computer  was 
easy  to  do. 

SD 

D 

A 

SA 

6.  Collaborating  with  my  team  using  the  computer  was 
just  as  effective  as  if  I  was  talking  to  them  face-to-face. 

SD 

D 

A 

SA 

7.  I  learned  a  lot  from  the  tutorial. 

SD 

D 

A 

SA 

8.  The  tutorial  prepared  me  well  to  solve  the  problem 
scenarios. 

SD 

D 

A 

SA 

9.  My  team  and  I  used  the  information  we  learned  on  the 
tutorial  to  solve  the  assessment  scenarios. 

SD 

D 

A 

SA 

10.  Overall,  the  tutorial  was  a  high-quality  product. 

SD 

D 

A 

SA 

1 1 .  I  liked  having  an  “animated”  cartoon  teach  the 
tutorial. 

SD 

D 

A 

SA 

12.  The  tutorial’s  graphics  and  animations  helped  me  to 
understand  difficult  concepts. 

SD 

D 

A 

SA 

13.  The  time  allotted  for  the  tutorial  was  just  about  right. 

SD 

D 

A 

SA 

14.  My  team  and  I  had  enough  time  to  solve  all  the 
problem  scenarios. 

SD 

D 

A 

SA 

15.  The  time  allotted  for  the  entire  program  (tutorial  and 
assessment)  was  just  about  right. 

SD 

D 

A 

SA 

16.  The  scenarios  in  this  program  were  realistic  and 
applicable  to  a  future  Air  Force  officer. 

SD 

D 

A 

SA 

17. 1  feel  better  prepared  to  solve  everyday  problems 
after  doing  this  program. 

SD 

D 

A 

SA 

18.  The  problem  solving  skills  I  learned  in  this  program 
will  help  me  in  my  Air  Force  career. 

SD 

D 

A 

SA 

ADDITIONAL  COMMENTS  ON  NEXT  PAGE 


19.  When  solving  complex  problems,  I  prefer  to  work: 

a.  By  myself. 

b.  With  one  partner. 

c.  With  two  partners. 

d.  With  more  than  two  partners. 

20.  Please  write  down  the  general  steps  that  you  followed  to  solve  the  assessment 
problem  scenarios. 


21 .  What  did  you  like  best  about  the  Problem  Solving  program? 


22.  What  would  you  do  to  make  this  program  better? 


APPENDIX  F 

PRE-TREATMENT  INSTRUMENT 


PROBLEM  SCENARIO 

“ You  are  the  captain  of  your  high  school  soccer  team  and  you  have  noticed 
that  the  younger  members  of  the  squad  are  being  disrespectful  to  the  coach 
behind  her  back.  This  attitude  is  starting  to  affect  the  discipline  of  the  team 
and  the  performance  on  the  field...” 

In  the  space  below,  briefly  explain  the  PROCESS  you  would  follow  to  solve  the 
problem.  (NOTE:  Do  not  provide  a  solution;  just  describe  the  process  you  would 
use.) 


APPENDIX  G 
RECRUITMENT  SCRIPT 


CLASSROOM  RECRUITMENT  SCRIPT 

“GOOD  MORNING  (AFTERNOON).  MY  NAME  IS  DANIEL  URIBE 
AND  I  AM  CURRENTLY  A  STUDENT  AT  ARIZONA  STATE 
UNIVERSITY  PURSUING  A  PHD  IN  EDUCATIONAL  TECHNOLOGY. 
THE  PURPOSE  OF  MY  VISIT  TODAY  IS  TO  ASK  FOR  VOLUNTEERS 
TO  PARTICIPATE  IN  A  STUDY  THAT  I  WILL  BE  CONDUCTING 
OVER  THE  NEXT  TWO  LESSONS.  THE  PURPOSE  OF  THE  STUDY  IS 
TO  INVESTIGATE  THE  EFFECTIVENESS  OF  TWO  DIFFERENT 
TEACHING  METHODS  ON  THE  ABILITY  OF  STUDENTS  TO 
RESOLVE  ILL-DEFINED  PROBLEMS  IN  A  WEB-BASED 
ENVIRONMENT. 

THE  STUDY  WILL  TAKE  PLACE  DURING  TWO  CLASS  PERIODS 
AND  WILL  REQUIRE  THAT  YOU  GO  TO  THE  LANGUAGE 
LEARNING  CENTER  (LCC)  TWICE.  AT  THE  LCC  YOU  WILL  FIRST 
WORK  THROUGH  AN  ON-LINE  TUTORIAL  ON  PROBLEM 
SOLVING.  AFTER  THE  TUTORIAL  YOU  AND  YOUR  TEAMMATES 
WILL  BE  GIVEN  SEVERAL  PROBLEM  SCENARIOS  TO  SOLVE 
WHILE  COMMUNICATING  ONLINE.  AT  THE  END  OF  THE  STUDY, 
YOU  WILL  ALSO  BE  COMPLETING  A  SHORT  SURVEY  ABOUT  THE 
PROGRAM. 

IT  IS  IMPORTANT  FOR  YOU  TO  UNDERSTAND  THAT 
PARTICIPATION  IN  THIS  STUDY  IS  STRICTLY  VOLUNTARY.  YOU 
WILL  NOT  BE  PENALIZED  IN  ANY  WAY  FOR  NOT  PARTICIPATING 
AND  YOU  CAN  CHOOSE  TO  CEASE  YOUR  PARTICIPATION  AT 
ANY  TIME  WITHOUT  RETRIBUTION.  IF  YOU  DECIDE  TO 
PARTICIPATE  THE  TOP  TEAM  IN  EACH  GROUP  AND  THE  TOP 
INDIVIDUAL  SCORER  WILL  BE  TREATED  TO  DINNER  AT  A 
POPULAR  LOCAL  DINING  ESTABLISHMENT.  ARE  THERE  ANY 
QUESTIONS? 

I  WILL  NOW  DISTRIBUTE  THE  INFORMED  CONSENT  FORMS  AND 
THE  PARTICIPANT’S  BILL  OF  RIGHTS.  PLASE  READ  THE  BILL  OF 
RIGHTS  AND  THE  INFORMED  CONSENT  FORM  CAREFULLY.  IF 
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YOU  AGREE  TO  PARTICIPATE  IN  THIS  STUDY  PLEASE  SIGN  THE 
INFORMED  CONSENT  FORM  AND  PASS  IT  FORWARD.’’ 

[PAUSE  WHILE  THE  CADETS  READ  THE  FORMS  AND  SIGN  THE 
ICD] 

“THOSE  OF  YOU  WHO  AGREED  TO  PARTICIPATE  WILL  RECEIVE 
A  COPY  OF  THE  INFORMED  CONSENT  FORM  WITH  ALL  OF  THE 
APPROPRIATE  SIGNATURES  PRIOR  TO  THE  START  OF  THE 
STUDY.  THANK  YOU  FOR  YOUR  TIME  THIS  MORNING 
(AFTERNOON)  AND  HAVE  A  GOOD  DAY.” 


APPENDIX  H 

SUBJECT’S  BILL  OF  RIGHTS 
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Research  Subject’s  Bill  of  Rights1 

The  USAFA  Institutional  Review  Board  believes  that  personal  concern  for  each  subject’s 
welfare  is  indispensable  to  the  quest  for  knowledge.  The  most  important  person  in 
medical  or  behavioral  research  is  the  subject.  The  research  subject  is  the  essential 
element  without  whom  health,  disease  or  behavior  could  not  be  observed  and  response  to 
treatment  or  situations  could  not  be  measured. 

The  obligation  to  protect  human  subjects  applies  to  research  conducted  using  Department 
Of  Defense  (DOD)  facilities  or  property,  supported  with  DOD  funds,  or  performed  by 
DOD  employees  or  contractors.  If  you  are  asked  to  be  a  research  participant,  you  should 
know  the  requirements  for  protecting  your  rights  to  information,  privacy  and  well-being. 
The  rights  below  are  the  rights  of  every  person  who  is  asked  to  be  in  a  research  study. 

The  right  to 

1 .  considerate  and  respectful  treatment  as  a  research  subject  and  information  on  who  to 
contact  if  I  think  I  am  not  being  treated  appropriately. 

2.  know,  by  name,  the  researcher/care  provider  responsible  for  coordinating  my 
activities/care. 

3.  be  told  how  and  why  I  was  chosen  for  the  study. 

4.  be  told  what  the  study  is  trying  to  find  out. 

5.  be  told  what  will  happen  to  me  and  whether  any  of  the  procedures,  drugs,  or  devices 
is  different  from  what  would  be  used  in  standard  practice. 

6.  be  told  about  the  frequent  and/or  important  risks,  side  effects,  inconveniences,  or 
discomforts  resulting  from  the  things  that  will  happen  to  me  for  research  purposes. 

7.  be  told  of  other  choices  I  have  and  how  they  may  be  better  or  worse  than  being  in 
the  study. 

8.  be  told  if  I  can  expect  any  benefit  from  participating,  and,  if  so,  what  the  benefit 
might  be. 

9.  be  allowed  to  ask  any  questions  before  agreeing  to  be  involved  and  then  during  the 
course  of  the  study. 


1  This  bill  of  rights  has  been  adapted  from  similar  documents  developed  by  the  American  Hospital 
Association,  the  University  of  California  at  San  Francisco  and  the  Office  of  Energy  Research. 


10.  refuse  to  participate  at  all  or  to  change  my  mind  about  participation  after  the  study  is 
started  and  to  be  told  who  to  contact  to  terminate  my  participation.  This  decision 
will  not  affect  my  right  to  receive  the  care  I  would  receive  if  I  were  not  in  the  study. 

1 1.  be  free  of  pressure  when  considering  whether  I  wish  to  agree  to  be  in  the  study  or 
where  to  get  help  if  I  think  l  am  being  pressured  to  participate  in  a  study. 

12.  be  told  what  sort  of  medical  treatment  is  available  if  any  complications  arise. 

13.  expect  that  all  communications  and  records  pertaining  to  this  research  will  be 
treated  as  confidential  to  the  extent  permitted  by  law. 

14.  know  in  advance  how  much  time  commitment  is  necessary  for  involvement  in 
research. 

15.  receive  a  copy  of  the  signed  and  dated  consent  form. 


If  I  have  other  questions  I  should  ask  the  researcher.  In  addition,  I  may  contact  the 
USAFA  Institutional  Review  Board  (IRB),  which  is  concerned  with  protection  of 
volunteers  in  research  projects,  at  (719)  333-2587. 


APPENDIX  I 

INFORMED  CONSENT  DOCUMENT 


INFORMED  CONSENT  DOCUMENT 


DEPARTMENT  OF  THE  AIR  FORCE 

Department  of  Foreign  Languages 
USAF  ACADEMY,  COLORADO,  80840 


Privacy  Act  and  Freedom  of  Information  Act 

I  understand  that  records  of  my  participation  in  this  study  may  only  be  released  in  accordance  with  federal  law;  The  Freedom 
of  information  Act,  5  U.S.C.  552,  the  Federal  Privacy  Act,  5  U.S.C.  652a,  and  their  implementing  regulations  niay  apply. 


TITLE  OF  STUDY 

The  effect  of  Instructional  method  and  computer-mediated  collaboration  oh  performance  resolving  iff-defined  problems. 

PROTOCOL  NUMBER  FAC2O02O13  DATE  STUDY  APPROVED  28  June  02  DATE  ICO  APPROVE  28  June  02 


INVESTIGATORS'  NAME(S),  DEPARTMENTS),  PHONE  NUMBER(S) 

LI  Col  Richard  Sutherland,  DFF,  x3820 

Maj  Dan  Uribe,  PhD  Candidate,  Arizona  State  University,  (480)  750-4786 


PURPOSE  OF  STUDY 

I  understand  that  I  am  being  asked  to  participate  in  a  research  study.  The  purpose  of  the  study  is  to  determine  if  teaching 
students  how  to  solve  problems  systematically  (i.e.  step-by-step)  is  any  better  than  teaching  students  how  to  solve  problems 
through  the  use  of  case  studies.  There  will  be  approximately  200  cadets  participating  in  the  study,  which  will  take  place  over 
two  50-mtnute  class  periods. 


PROCEDURES 

Approximately  1  week  prior  to  the  study,  I  will  receive  instructions  on  howto  log-in  to  the  Blackboard  system.  At  that  time  I 
will  also  receive  general  instructions  on  where  to  go  on  the  day  of  the  study.  On  the  day  of  the  study,  I  will  be  directed  to  a 
workstation  in  the  Language  Learning  Center  and  will  follow  the  instructions  that  will  be  displayed  on  the  screen  At  the  end  of 
the  computer-mediated  program, !  will  complete  an  attitude  survey  to  collect  information  about  my  attitudes  toward  the 
instructional  program,  working  in  groups  and  transfer  of  the  information  learned.  My  personal  information  will  be  stored  In  the 
DFF  server  and  will  be  password  protected.  This  personal  information  will  only  be  accessed  by  the  primary  and  associate 
researchers.  Any  identifiable  information  wil!  be  destroyed  once  the  data  analysis  is  complete. 


BENEFITS 

i  understand  there  are  direct  benefits  from  my  participation  in  this  study.  If  I  am  a  member  of  one  of  the  top  teams  or  if  I  am 
the  top  individual  scorer,  I  understand  I  will  be  treated  to  dinner  at  a  popular  local  dining  establishment. 

I  understand  that  an  alternative  to  participation  in  this  research  is  to  not  participate. 


RISKS/INCONVENIENCES 

I  understand  there  are  no  risks  associated  with  this  study.  I  also  understand  that  the  only  inconvenience  for  participating  in 
this  study  will  be  the  two  trips  I  will  have  to  make  to  the  Language  Learning  Center. 


COMPENSATION  FOR  TREATMENT  OF  INJURY 

I  understand  that  my  entitlement  to  medical  and  dental  care  and/or  compensation  in  the  event  of  injury  is  governed  by  federal 
laws  and  regulations.  If  I  have  questions  about  my  rights  or  if  I  believe  I  have  received  a  research-related  injury,  I  may 
contact  the  USAF  Academy  Institutional  Research  and  Assessment  Division  (HQ  USAFA/XPR)  at  719-333-2587. 
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DECISION  TO  PARTICIPATE 

The  decision  to  participate  in  this  study  is  completely  voluntary  on  my  part  No  one  has  coerced  or  intimidated  me  into 
participating  in  this  program.  My  alternative  to  participating  In  this  study  is  to  choose  not  to  participate.  I  understand  that  if  I 
refuse  to  participate,  I  will  not  lose  any  benefits  that  I  am  entitled  to.  I  also  understand  that  a  non-graded  activity  will  be 
available  for  me  as  an  alternative  to  participation  in  this  study.  I  am  participating  because  I  want  to.  My  investigator  has 
adequately  answered  any  and  all  questions  I  have  about  this  study,  my  participation,  and  the  procedures  involved.  I 
understand  that  an  investigator  will  be  available  to  answer  any  questions  concerning  procedures  throughout  this  study.  I 
understand  that  if  significant  new  findings  develop  during  the  course  of  this  study  that  may  relate  to  my  decision  to  continue 
participation,  I  will  be  informed.  I  further  understand  that  I  may  withdraw  this  consent  at  any  time  and  discontinue  further 
participation  in  this  study  without  prejudice  to  my  rights.  I  also  understand  that  the  investigator  may  terminate  my  participation 
in  this  study  at  any  time  if  he/she  feels  this  to  be  in  my  best  interest.  I  have  been  provided  a  copy  of  this  consent  form. 


QUESTIONS  REGARDING  MY  PARTICIPATION  IN  THIS  RESEARCH  STUDY 

If  I  have  questions  about  this  research  study,  I  should  contact  the  principal  investigator,  Lt  Col  Richard  Sutherland,  at  x3820. 
If  I  have  questions  about  my  rights  as  a  research  participant  or  if  I  have  received  a  research-related  injury,  I  should  contact 
the  USAF  Academy  Institutional  Research  and  Assessment  Division 
(HQ  USAFA/XPR)  at  719-333-2587. 


My  signature  below  indicates  my  willingness  to  participate  In  this  research  study,  my  receipt  of  a  copy  of  this 
document,  with  all  three  required  signatures,  and  a  copy  of  the  Research  Subject’s  Bill  of  Rights. 


Participant’s  printed  name 


Participant's  SSAN 


Participant's  signature 


Advising  Investigator’s  Signature  Advising  Investigator's  SSAN  Date 


I  witnessed  the  participant’s  signature  to  this  informed  consent  document. 


Witness’  Signature 


Witness’s  SSAN 


Distribution:  Principal  investigator  and  Participant. 
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